Standup, filed the 3rd
Q3 Migration: A Personal Accounting
The cutover was scheduled for a Tuesday. I would like to talk about the Tuesday.
Wen-Ju Halloran
Staff Engineer, Platform. Filed 3 September, published 6 September. Eleven minutes.
The migration was scheduled for the third. It did not happen on the third. What follows is an account of the week as I experienced it, filed under the understanding that my team will read roughly the first four paragraphs and then look for the part where I say a date.
I want to be careful here, because the version of this story I told in the channel on Wednesday was shorter and, in retrospect, more flattering. The desk has asked me to slow down and show the reasoning. So I am going to show the reasoning, and the reasoning is going to take eleven minutes.
What actually happened
We had two clusters, and we had a plan to have one cluster. The plan assumed that the read replicas would drain on their own schedule, which they did, and that the write path would follow, which it did not. Somewhere around four in the afternoon my time, which is the middle of the night for two thirds of the people who could have told me this was going to happen, the write path began queueing instead of failing.
Queueing instead of failing is the worst available outcome, because nothing alerts and everything is wrong. The dashboards were green for six hours. I would like the record to show that the dashboards being green is not the same as the system being fine, and that I have now said this in writing three quarters running.
By the morning we had a backlog of about nine hours and a decision to make about whether to replay it in order. We replayed it in order. That took the rest of Thursday, which is the reason there was no standup on Thursday, which is the reason I am filing this at eleven hundred words instead of two hundred.
Blockers
Three things are still open, and two of them are other people's, which I say without accusation and with a certain amount of accusation.
The first is that the write path still queues silently under load. We have a ticket. The ticket has been in the backlog since April and has been groomed four times, each time successfully, in the sense that it remained in the backlog.
The second is that I do not have the credentials to roll back the schema myself, which means every rollback goes through a person in a timezone eight hours ahead of me who is, correctly, asleep when I need them. I have asked for the credentials twice. I have been told twice that this is a process question.
The third is mine. I did not read the runbook closely enough before the cutover, and the runbook did say something about the write path, in a note, at the bottom, under a heading about something else.
Members only
The blockers are behind the paywall.
Field Notes gives every reader three free pieces a month. You have opened two. Blockers, incident notes and anything filed under a Friday count toward the same allowance.
What I need
A decision on the credentials by Friday, and someone other than me to own the ticket about silent queueing. I will file again on the tenth. If the write path does the same thing before then, the piece will be shorter and the tone will be different.
Second pass, returned to the author 5 September
The migration was scheduled for the third. It did not happen on the third. What follows is a full and unsparing account of the catastrophic weekWhat follows is an account of the week as I experienced it, filed under the understanding that my team will read roughly the first four paragraphs.
Queueing instead of failing is the worst available outcome, because nothing alerts and everything is wrong. Anyone who looked at the dashboards would have known this.The dashboards were green for six hours.
I have asked for the credentials twice. I have been told twice that this is a stalling tactica process question.
Held for the 6th. Slot confirmed. Kill fee not applicable.
Ottoline Brask
Engineering Manager, Platform
Good piece. I have sealed it twice, which is four percent of my quarterly allocation, so please understand what that means. Credentials question goes to the security desk today.
Devin Aumaugher
Staff Engineer, currently in Lisbon
Paragraph nine is the actual standup. Everything before it is context. I say this with affection and I say it every week.
Marguerite Okonjo-Beale
Senior Manager of Asynchronous Narrative
Flagging that this was filed on the 3rd and published on the 6th, and the thing it describes was resolved on the 4th. That is working as designed. I am noting it so that nobody notes it later.
Priya Ravikumar-Stagg
Interim Head of Remote Culture
I have added this to the onboarding reading list. New starters now read four standups before their first standup.
Responses over 300 words are treated as submissions and enter the queue.