So the warehouse crashed....

Completely. I mean if it were a cartoon it would have it's tongue hanging out the side of it's mouth and X's over the eyes. The DBA's called in UNIX and Oracle support guys almost immediately. Each blamed the other. Either way, my users had no data which tends to make them grumpy. After much ado, everyone agreed to blame the hardware since they were outnumbered.

It took two days to move everything over to another box on another frame while the original frame hardware was swapped out. My job of late has been to reassure people that yes, we know about the problem and yes, someone is working on it. I've referred a great many users to my boss since that's what he gets paid for.

One key event: A callout email/page went out around 3:30 AM on Wednesday morning. Oddly, my phone didn't ring so I didn't find out about the complete failure of our critical system until 7:00 when I logged on. I asked my boss why I wasn't called (not angry mind you, just confused). His response: "Because I'm too cheap to pay you for callouts". Ha! Take that you non-consultants!

My favorite moments are the users who name drop in hopes of resolution. Example:

Me: "Blah Blah Blah intro, how can I help you?"
User: "my web page is not working and I can't run reports."
Me: "That's correct. We have a global outage. Won't be back online until tomorrow"
User: "That's no good. I need to run a report for [Name] now."
Me (thinking): "Oh, it's for [Name]? Well why didn't you say so, let me just get my magic wand here..."
Me (what I actually said): "There's nothing more that we can do to make things go faster"

Same user calls back later....

User: Our repositories are local machines, why can't I run reports
Me: Because the database they're trying to hit is down.
User: Unacceptable. I need a local copy of that database installed on site so this doesn't happen again.
Me: OK. Fine by me.
User: (surprised) Really?
Me: Sure. If you want to create a backup site for the datawarehouse I have no objections. In fact, it would be great for resiliency.
User: Would that be expensive?
Me: It's going to cost many many thousands of dollars?
User: Why? We have servers here we can use
Me: The warehouse is many many terrabytes of data
User: Well if it's going to cost any more than $10,000...
Me: Well, call my boss, he's the one that's going to help you with that.

Comments

Post a Comment

Popular posts from this blog

For Gerard

 Many moons ago, I started reading blogs and shortly thereafter I started reading them.  I became discouraged because I read many writers, far more talented than me and the ease with which they elucidated the very points I wanted to make but so much more elegantly and completely that I just gave up.    Gerard of American Digest was one such writer.  His facility with prose and his ability to evoke emotion was remarkable.  I linked to him once with my nascent blog and he was kind enough to reach out to me and thank me for the link.  I literally had zero readers.  Just like now.  But that was the measure of the man.  He took the time to reach out to me and genuinely thank me for the link.   The closest I ever had to meeting the man was meeting Morgan in person.  Morgan and Gerard were friends.  They had an evident mutual respect for their respective works.   I never met Gerard, didn't know the man personally but having read his work I had a sense of the man.  He's dying now.  If
Read more

So....the autism thing

Since April is nearly over and it's autism awareness month, I suppose I should talk a bit about it. I guess the best way to start is to rehash the story of How I Got Here. My oldest was born in 1999. Healthy, happy and so on. One peculiar thing he did as an infant was to avoid eye contact. Our doctor assured us it was because he was so young and his eyesight was undeveloped and he was probably focusing on something in the background that was more interesting. Besides, babies don't recognize faces for a few months after birth. Only later did we find out that our doctor was an idiot. He was 16 mos. old when his little brother was born and he was walking and making wishes known by pulling me or my wife by the hand to whatever he wanted etc. He was fascinated with the wheels on toys. If you gave him a truck he'd turn it over and spin the wheels and smile. Only later would I find out this is a major warning sign. This was our first born and what the hell did I know?
Read more