More fun with the filestore

So last week I was telling you about the changes to the servers here at work. (Riveting reading, I know) The thing is, I'm the admin on a legacy app. This is a nightmare. It means that when it comes to maintenance we're not mission critical. Not until something goes wrong that is. We had to move from one server cluster to another. (These are virtual nodes) Normally, sane operators create the new system in parallel and then redirect from old server to new. They then wait to make sure everything goes well before decommissioning the old one. Nah, let's not do that. Let's just image the old server on the new one and then cut over. The redirect will work fine. No need to wait until after we've had the users try and hit the new server. We'll retire it over the weekend and wipe the old server clean. Folder permission, and all other settings will be lost but really, who cares?

Sunday night ~ 11:00 PM, log on and find that I cannot remote into the new server. Nor can the admin console of the tool I use find any server whatsoever. Call No Help Whatsoever Desk and they tell me it's a local setting. Impossible say I. I can ping but not remote into it. It's in the proper domain but no services can hit it. After much explanation I give up and pack it in. Monday morning at 3:30 the overnight jobs begin. Naturally, they fail. There's no destination server to write the jobs to nor is there even a filestore to fetch said jobs from. Epic fail. By 6:39 AM all jobs have failed and the very early risers are already emailing me and leaving VM messages. They're not happy. I explain the situation and that helps not one whit. Lather, rinse, repeat. I contact my boss who's not happy. I continue to call Server Nerds who immediately suggest that it must be me and my users. There's no way they could have made a mistake and after all, they've tested it and vouched for it.

Finally one of my users gives up and call one of the top 10 people in the company. That he is involved means klaxons all over the universe start going off. AAAAAAOOOOOOOOOGAAAAAAH! You know the sound from old naval movies.

Ninteen seconds after he calls and we talk to him we get the Lead Server Nerd on the phone who checks the registry and lo and behold, the redirect was REM'd out. Oops. Sorry Big Boss Man.

Restart morning batch ~ 10:30 AM and now everyone hates me because I was telling them last week that this whole thing was going to go pear shaped. Despite the fact that I have no ownership of said server clusters makes me the bad guy 'cause I saw this coming.

Moral of the story: The redirect from old server to new never works the first time. Always check the registry. After you've done that, check it again.

Comments

Eh.....they'll get over it.
The Last Ephor said…
You'd think so, but no. This is still going on. We had to resurrect the old server and get the registry values and settings.

Popular posts from this blog

So....the autism thing

For Gerard