Over on the SQL Server Central blog, Steve Jones has a blog about, and call for, stories of the worst day on the job. That included, for him, a 30 hour stint after a failed cutover to SQL Server, which is a nice reminder of why I never wanted to be a DBA. All programmers are different.
Production problems like these are the bedrock of programming bad days. Programmers love to share their war stories. Our solutions architect Josh recounted his: going live with a mobile app change that, in the process of sending updates to users, deleted their activity. In his case, the frustration was compounded by the fact that he knew the problem could occur, but was forced to push it to production anyway.
My own worst day happened when I accidentally deleted a subset of production data by running my first database update program. To my credit, the code had been blessed by a more senior programmer, who clearly did not catch what was a simple bug.
Nonetheless, I did destroy data. Fortunately, a backup of the database had been run only a few hours earlier, so we were able to restore without any significant interruption.
I’ve dealt with worst production problems as I know many other programmers have. The kind that wake you up at 2 in the morning, or that require extra hours (although thankfully nothing like 30 hours) and lots of one-off programs to resolve. But there is a reason that that particular production problem ranks as my worst day on the job – it was my error that caused it.
I think this is a fundamental truth about production errors – having a problem that impacts your organization’s production systems is a nightmare… unless you didn’t actually cause the problem. Then it gets interesting.
Monkeyuser nails it in 5 panels.
The quality that sets programmers apart from others at their company is that they have the understanding of underlying systems and the tools to fix them. Patching a production problem is a confirmation of that you have valuable skills to offer your organization – skills that the c-suite may not realize or value; that is, until there is a crisis.
It’s also really fun to figure out a solution, even if there is a bit of stress connected with it. There is a sense of camaraderie that comes from coming up with a solution (or workaround) under pressure, and it’s gratifying to learn how much you can rely on your team members in a crisis. I was fortunate in the teammates with whom I worked. I never found there was time for blame during one of these incidents, just everyone working together to solve a problem.
