Server Room Craziness

A funny post over at The Daily WTF reminded me of a situation I witnessed many years ago at a small client I was working for.

Like the situation described in the post, the client needed to move the door to the server room.  I can’t even remember why but they were moving the door from one wall to another wall.

On the morning that the work was supposed to begin, the crew that was working on removing the door and walling in the opening (there were separate work crews for creating the new opening and closing the old) showed up early and removed the door and closed the opening before anyone arrived to work.  As luck would have it, one of the production servers locked up and needed to be hard rebooted.  The problem was there was no longer a door to the server room.  After considerable debate about the matter, the admin ended up having to actually chop a hole in the wall to the server room with a crowbar that he retrieved from the trunk of his car.

Having a Bad Day Video

This link comes courtesy of Scott Riggins.  I like the part where the guy jumps over the conference table and starts beating on the guy who was talking too loud on the phone.  It reminds me of an instance recently when I was in an office where two people just down the hall from each other were talking on speaker phones (to each other).  Since I was about in the middle, I had the pleasure of listening to both conversations live out of their mouths and then echoed back a quarter second later over the speaker phone.  Sort of a perfect storm of office annoyance.

The Tuesday Night Upgrade

This is number 7 in my series of ridiculous happenings in my software development career.

I had been at Tiny Trader for less than a month when the CTO called a meeting to discuss a scheduled upgrade to the accounting system.  The plan was that the MS SQLServer instance would be upgraded and the new version of the software would be rolled out to the users.  The CTO asked me and one of my colleagues to watch over the consultants being brought in to perform the upgrade.

Everything seemed strait forward enough until the CTO explained that the upgrade would be taking place on a Tuesday evening.  After explaining that it is never a good idea to perform a major system upgrade during the week, everyone assured me that I had nothing to worry about because these consultants were "highly qualified" and had estimated only a few hours for the upgrade.

Against my advice, everything proceeded as planned.  On the evening of the upgrade, the consultants arrived and introduced themselves as Moe and Larry.  Moe was older and clearly in charge.  Larry was even younger than I was at the time and was obviously the technical person.

Moe and Larry were given the login credentials to the SQLServer machine and began the upgrade by sliding in the CD.  Within 5 minutes, Moe and Larry were looking at each other in horror and discussing what to do.  My colleague and I stepped up to the monitor and observed that the upgrade had failed in craptacular fashion, in mid stream, because a file was locked.  After minimizing a few windows, I discovered that Moe and Larry had not shutdown the database service.

Just so we all are on the same page, let me give you a little background on the technology in use at this client.  The server machine was a DEC Alpha running Windows NT 3.5 and SQLServer 4.2.  The  DEC Alpha part becomes important much later in the night.  For those fortunate enough not to have lived through this stage in Windows history, Windows NT actually ran on processors that were not Intel compatible.  They were not very popular but every now and then you would run into one.  Windows NT 3.5 pretty much assumed that you knew what you were doing when you installed stuff.  Installers back then also assumed you were smart enough to shutdown software before upgrading it so there was no rollback or anything like that when the install failed.

The rest of the night, or at least what those losers could stay awake for, Moe and Larry sat back while my colleague and I were at the keyboard, on the phone with Microsoft. Microsoft had good news for us.  There were utilities that Microsoft had written to help with situations like this.  The utilities were able to recover partially upgraded data files.  The utilities only worked on data files that were generated on Intel compatible platforms. 

We ended up having to hand edit some of the data files so that SQLServer would recognize them again.

Over 11 hours later, the SQLServer instance was back up and running, just in time for the accountants to come in to work on Wednesday morning.  We even upgraded it. 

Moe and Larry were long gone and never invited back for a return visit.

Tags:

Just Reboot It!

This is number 6 in my series of ridiculous happenings in my software development career.

I had left Upstart Consulting and gone out in to the world as an independent consultant.  My first project was fairly uneventful but I had just started a new project for a small energy trading company that I will call Tiny Trader.

I came to work at Tiny because of a relationship I had with their CIO from my years at Upstart.  She called me in almost a state of desperation and I could not say no.  She hired both me and a close friend and co-worker of mine without even an interview.

On my first day, I was given a tour of the operations of Tiny Trader.  During the 2 hour tour, I overheard announcements 3 times for all users to exit an application while the database was being repaired.  A quick explanation confirmed my suspicions that the entire trading business of Tiny was run on MS Access and that several times a day the database was corrupted and had to be repaired.

As we entered the server room, I could not help but notice the huge pile of network cabling jumbled on the floor.  From a patch panel above, wires led to the pile and from the pile back to the panel.  As we looked around, a user entered the room and walked up to a server.  The user rebooted the server and walked out.  The administrator explained that all accounting system users were able to login to the database server with admin privileges and often rebooted the box when they were encountering unexplained problems.

A brief question and answer session revealed that no database dumps were occurring and even though backups were scheduled on the server daily, the data was obviously not backed up because of this.  The administrator was oblivious to this important fact. 

The good news was that there were no processes in place to hinder improvement.  That bad news was the same as the good news.

On the second day of work at Tiny Trader, two users got into a fist fight in the conference room and broke the glass cover for the conference table.  This was definitely going to be an interesting project.

Tags:

July or Die!

This is number 5 in my series of ridiculous happenings in my software development career.

The project for the railroad company was now going full steam ahead but it was becoming quite clear that initial estimates were not close to reality.  Interesting technology choices were made like writing the back end processes in COBOL even though the system was to be deployed on HP Unix boxes.  Rational arguments as to why this was not a very good idea were frowned upon.

As moral continued to dip, management needed some way to motivate the troops for a final push to rescue the project from cancellation.  A deadline of July 1 was selected as the date for code completion.  To this day, I have no idea how this particular date was chosen.  July 1 certainly had no basis in reality.  Everyone on the project, even the management I am certain, had no faith in our ability to deliver in this time frame.  Nonetheless, we were officially on a death march and I had little say in the matter.

Everyone was gathered in a large conference room for the big announcement of the new deadline and notification of vacation cancellations and mandatory 60 hour weeks.  You know, the sort of motivational speech we all look forward to hearing.

As we entered the room, each person was given a small box.  We were instructed not to open the box until the crucial moment in the presentation when we would be told to do so.

When the moment arrived, we all opened the boxes to find a new coffee mug.  I was so moved by this mug that I have kept it to this day.



Management never figured out why the mugs seemed to have a negative impact to moral.

Tags:

The Toilet Bowl Manager

This is number 4 in my series of ridiculous happenings in my software development career.

I had just left my first paying job at Ubber-IT Company to start a new job at a small consulting firm specializing in the new paradigm of "Client-Server" development.  I will call this company Upstart Consulting.

Upstart sent me to several weeks of training on their methodology and development tools of choice and then sent me out on my first project.  The project was for a large railroad and our task was develop a scheduling and settlements system for their railcar transactions.

After being at the client site for a few weeks, I dropped by the men’s room one day after lunch for a quick visit.  As I entered the room, I heard an unusual noise that I could not identify.  I looked around and saw a hand drop down from under the first stall wall.  I thought to myself, "My god, this guy is having a heart attack or something!".

I rushed to the stall door and looked over the top.  The guy was sleeping on the toilet!  Snoring quite loudly, I might add.

I went back to my team and told them what I had just seen.  One of my co-workers, an employee of the client, said "Oh yeah, that is <name removed>, he is a manager over the <group name removed> area.  He sleeps on the toilet every day after lunch." 

A brief survey of his employees and peers revealed they were all aware of this interesting habit.

At that point in my career, I had not yet been exposed to the enormous amount of waste that goes on in large companies and I have not seen a situation of this magnitude of acceptance since then. 

Apparently, this goes on much more than my experience tells me.  Here is a video that is very funny.

By the way, Toilet Bowl Manager’s technique was very similar to Coma in toilet 1.

Tags:

The Marathon Man

This is the third of my series of ridiculous people and happenings that I have come across in my career.  If you have been following along, you know that my career (at least paid career) started at a large company I refer to as Ubber-IT Company.  My first role at Ubber was in a support group working with jcl mostly but generally performing the tasks that nobody else wanted to deal with.

I had moved into a new role with a group that developed document imaging systems using that crazy new technology paradigm called "Client-Server".  I enjoyed working in this group because everything new in tech was what we were working with.  What also made the role interesting was the group of developers I worked with.  They were real programmers.  I have never had the chance to thank this group of developers for the inspiration they gave me, but I am convinced that I would never have accomplished what I have without working early on with a group like that.

However, my group back then is not the focus of the discussion today.  My team members and I sat in a huge cube farm and a few rows over was a group that worked on a project for Huge Government Agency.  This group was well known in Ubber for working obscenely long hours.  Calling what they did a death march doesn’t seem to capture it well because a march implies that it will end at some point.

There was one individual in particular who seemed to go beyond what the others on the Huge Government Agency project did.  I will call him Marathon Man.

I have always been an early riser, getting in to the office around 6 am.  Back then, with no wife and family, I also stayed quite late.  On occasion, I would even drop by the office on weekends and sometimes very late.  Even at these odd hours, I do not recall every being at the office when Marathon Man was not working diligently in his cubicle.

I came to the office early one Monday morning to find that Marathon Man was not in his cube.  Next to his cube was a window that had plywood covering it.  I walked over and wires were exposed from his desk, where his tube and keyboard used to sit.  I looked out the window (we were on the second floor) and below were parts of what used to be Marathon Man’s terminal.  Ubber maintenance had attempted to clean up the mess but there were far too many small parts to get it all in one pass.

Nobody said anything about this.  A few days later, a new tube appeared in Marathon Man’s cubicle and a new window replaced the plywood.  Still no sign of him.

A few weeks later, I saw one of Marathon Man’s teammates in the coffee room and quietly asked what had happened.  He said, "The guy just snapped.  He had a bug that absolutely needed to be fixed by Monday morning and he wasn’t going to make it.  He finally tore his tube out and threw it out the window.  Nobody has heard from the guy since."

I walked out of the coffee room stunned at what I had heard.  I knew then that Ubber wasn’t the company for me and started planning my departure.

I never saw Marathon Man again.  I think of him every now and then and I envision him at a tropical resort, sipping a drink with umbrellas in it, watching the sunset.

Tags: , ,

The Smoke Stack

This is the second in my series on ridiculous things that have happened to me in my career.  As you may recall, I started off my career at a large IT company that I will call Ubber-IT Company  My position was in a support group for other development teams.  However, my team was not the only support team.  There was another team composed of only one guy as far as I could tell.  I will refrain from giving his real name.  Instead I will refer to him as The Smoke Stack.

The Smoke Stack was a system’s engineer in the real sense of the word.  His job was to actual write code that interacted with the mainframe OS.

Most people preferred not to interact with The Smoke Stack.  Back in those days, people were actually allowed to smoke inside some office buildings.  Even though smoking was still technically allowed, very few people smoked inside.  The Smoke Stack was one of these people and the Ubber gave him an enclosed office to reduce the chance of the offending odor engulfing the entire area.  In addition to the heavy smoking, he was also very offensive in his speech.  To make matters worse, he spoke broken English so you were never quite sure exactly what he said.

Needless to say, people avoided The Smoke Stack if at all possible and he seemed to enjoy that.

One day, The Smoke Stack needed a job created by my team and refused to give any written specifications to my manager.  The manager ordered to me to extract the specifications from The Smoke Stack and I was instructed to not perform any work until a suitable spec had been written.

I reluctantly went to Smoke Stack’s office and knocked on the door.  From inside I heard "@#%$# $#@@!  Come in!".  As I walked in the office, I was overcome by the smell.  The office was engulfed in a fog from the cigarette smoke and my eyes began to water.  There were stacks of dump output (core dumps from mainframe jobs) everywhere in the office and empty food containers stacked in the corner.  I’m certain that no member of the Ubber janitorial staff had entered this office in at least a decade.

There were five tubes (3270 emulator for you younger readers) on his desk generating quite a bit of heat for this small office.  Additionally, The Smoke Stack had a space heater generating at full capacity under his desk.  This made for an experience in heat usually reserved for the sauna when combined with the Ubber-IT Company requirement of wearing a suit with your jacket on at all times.

The first thing I said was "My manager is requiring a written spec before work on your job starts so I am here to get everything written down". 

The Smoke Stack countered with "That #@%$# @##@!.  I don’t write $@!$$ specs.  If you can’t understand what I $%#@$ say, and do it right, you shouldn’t be %$#&@ working here!".

At this point, I only been in his office for 45 seconds.  My eyes were bloodshot and I was sweating profusely.  I had to think of something fast or might end up in the corner with those food containers.  I decided to go with flattery.  "I understand where you are coming from.  It must be difficult working with these people for as long as you have.  I understand what you want but my manager absolutely will not allow me to work on this without something written down.  Can you help me out?".  Sort of a help me to help you moment.

The Smoke Stack looked at me, puzzled.  He mumbled "@#$%!" as he grabbed a stained napkin from his desk and quickly scribbled a few sentences on it.  The napkin appeared to have been previously used to wipe up a coffee spill.  At least that is what I made myself believe.

He handed me the spec/napkin and I was out of there, glad to get out of the office before passing out.  As far as I was concerned, that was the best spec I had ever seen.

You’ll Get a Lump of Coal and Like It

In the spirit of the upcoming holiday season, I thought I would create a series of postings on crazy, ridiculous, and downright scary things that have happened to me during my career as a software developer.  Generally, these are the types of things that you can only see if you have spent your career as a consultant.  It just takes that amount of moving around to see so much craziness.  Many of these situations are only funny because so much time has gone by and the pain seems to be forgotten.  These first of these stories is about the lump of coal.

My very first paid programming job was with a huge IT company that shall remain nameless in the post.  Rest assured, you know them but for our purposes we will call it Ubber-IT Company.  My job involved doing everything the experienced members of the team did not want to do.  Things like being on call 24 hours, making sure dataset compression jobs ran (yes, mainframe stuff), and running reports of the TPS variety.

My team was a support team and one of our customer groups was performing a very large, extremely complicated merge of computer systems between two companies that had undergone a corporate merger.  This team of people that I helped support worked absolutely obscene hours for many months.  I heard rumors of failed marriages, health problems, and other undesirable side effects from such a death march.

This project eventually concluded and the day came for the end of project celebration.  Ubber-IT Company was big on giving project trophies.  Everyone who had worked at Ubber for any length of time proudly displayed their project trophies on their cube desk like badges of a war hero and each team member was already clearing space for a new addition.  The VP of Ubber-IT Company came and gave a speech praising the team for their extraordinary level of effort and assured them that they had made a huge profit for Ubber.  Then the trophy boxes were brought out and each team member was announced by name and came forward to receive their badge of honor.

Later, I stopped by the desk of a friend of mine to see what the new trophy was like.  I looked over his trophy collection and could see no new additions.  Just as I had finished looking, he walked up and said, "Oh, are you looking for this?", as he pulled the box out of the garbage can.  He handed me the box and inside was a nice wooden platform with a lump of coal mounted on it.  Along with the coal, came a small piece of paper with a saying that was something to the effect of every diamond starts out as a lump of coal.

I always imagined that the janitors at Ubber-IT Company could not have been happy to carry all of that coal to the dumpster that night.

Tags: ,