The Tuesday Night Upgrade

This is number 7 in my series of ridiculous happenings in my software development career.

I had been at Tiny Trader for less than a month when the CTO called a meeting to discuss a scheduled upgrade to the accounting system.  The plan was that the MS SQLServer instance would be upgraded and the new version of the software would be rolled out to the users.  The CTO asked me and one of my colleagues to watch over the consultants being brought in to perform the upgrade.

Everything seemed strait forward enough until the CTO explained that the upgrade would be taking place on a Tuesday evening.  After explaining that it is never a good idea to perform a major system upgrade during the week, everyone assured me that I had nothing to worry about because these consultants were "highly qualified" and had estimated only a few hours for the upgrade.

Against my advice, everything proceeded as planned.  On the evening of the upgrade, the consultants arrived and introduced themselves as Moe and Larry.  Moe was older and clearly in charge.  Larry was even younger than I was at the time and was obviously the technical person.

Moe and Larry were given the login credentials to the SQLServer machine and began the upgrade by sliding in the CD.  Within 5 minutes, Moe and Larry were looking at each other in horror and discussing what to do.  My colleague and I stepped up to the monitor and observed that the upgrade had failed in craptacular fashion, in mid stream, because a file was locked.  After minimizing a few windows, I discovered that Moe and Larry had not shutdown the database service.

Just so we all are on the same page, let me give you a little background on the technology in use at this client.  The server machine was a DEC Alpha running Windows NT 3.5 and SQLServer 4.2.  The  DEC Alpha part becomes important much later in the night.  For those fortunate enough not to have lived through this stage in Windows history, Windows NT actually ran on processors that were not Intel compatible.  They were not very popular but every now and then you would run into one.  Windows NT 3.5 pretty much assumed that you knew what you were doing when you installed stuff.  Installers back then also assumed you were smart enough to shutdown software before upgrading it so there was no rollback or anything like that when the install failed.

The rest of the night, or at least what those losers could stay awake for, Moe and Larry sat back while my colleague and I were at the keyboard, on the phone with Microsoft. Microsoft had good news for us.  There were utilities that Microsoft had written to help with situations like this.  The utilities were able to recover partially upgraded data files.  The utilities only worked on data files that were generated on Intel compatible platforms. 

We ended up having to hand edit some of the data files so that SQLServer would recognize them again.

Over 11 hours later, the SQLServer instance was back up and running, just in time for the accountants to come in to work on Wednesday morning.  We even upgraded it. 

Moe and Larry were long gone and never invited back for a return visit.



Comments are closed.