Five lessons of a datacenter overhaul

Another important part of vendor watching is staying on top of equipment orders. We weren't nearly careful enough here. Don't just place the order, glance at the P.O., and assume they're shipping what you want. We did and it hurt. Even the best vendors with the best intentions can make critical mistakes when filling orders. Only the caution of Phil Rapoza, our facilities manager, saved us from APC's condenser spec-and-switch. We also had a full cable management system spec'd out and ordered, but suddenly the vendor (who shall remain nameless) backed out, claiming resource problems. Here again, Phil Rapoza and his band of merry men saved the day, fabricating cable ladders customised for the room when an alternative supplier couldn't be found in time.

Your project's problems might have different root causes , but in an industry that moves as fast as ours companies can go out of business, shift direction, or be acquired over the course of a weekend, leaving customers holding the bag when orders disappear into the ether. Count on orders and shipments to go wrong. Plan for the unexpected by getting an early jump when you can and building time for unexpected delays into your project schedule.

Lesson 5: Make a migration checklist and check it twice

Finally, moving day arrived. To make our migration easier, we'd contacted the boisterous folks at Silverback Migration Solutions, a Walnut Creek, CA-based outfit that specialises in helping companies perform datacenter migrations and build-outs. Where a general IT staff might take days to put racks together, add shelving and other accessories, slide the servers in on new rails, and test functionality end-to-end, Silverback cranks through these tasks in record time, sometimes installing 30 or 40 full racks of servers in a single day. (In our case, it was 10 racks in a few hours; see "Pimp my datacenter: SilverBack Migration Solutions".)

But while Silverback's on-site reps were willing, our planning was weak. Though we'd had months to do the prep work, we'd slipped into complacency and simply assumed certain things would work out as desired. Murphy gleefully proved us wrong.

Using APC's datacenter planning software, we'd created the necessary blueprint of our new physical layout, but instead of fleshing that out we let it go and assumed that auto-generated floor maps were enough. A conversation with Silverback and Rackwise reps cured us of that notion, sending us back to the drawing board to fill in some important gaps.

APC's rack maps were a good start, but they're not designed to take into account customized weights of individual pieces of infrastructure -- they use reference weights from a vendor database in order to provide ballpark figures. So the standard weight APC provides for a Dell PowerEdge 1650, for example, might reflect a configuration with two hard drives, whereas our servers might have four. Not a big difference for one server, but when you multiply by dozens per rack, and you're facing an 800-pound weight limit, the true weight becomes important. We were forced to make several rack reconfiguration decisions on the fly.

A second important omission was failing to gather full technical documentation for the equipment being migrated. Because the HIG 319 datacenter was to serve as a co-location facility for a number of SOEST departments, we would be re-racking, rewiring, and reassembling systems from all around campus. Detailed notes -- and aha, admin passwords! -- would be needed to put them all back together again. Yes, we not only lacked detailed information on how some of that equipment was configured, we even failed to collect the admin passwords for six servers we moved from another building. That meant we couldn't bring them up for testing until their research administrators could be found. Like most server migrations, ours was performed during off-hours, so not having the passwords wound up pushing the final equipment testing part of our plan well into production hours on the following day.

A migration day typically leaves little room for error or indecision. Before that day comes, you should have a punch list -- a list of detailed, step by step instructions -- that will guide everyone's actions from start to completion. We suggest that all team members keep notes on loose ends and to do's, and provide them to the team leader in time to create the punch list before any vendors go home. Don't let your solutions providers leave unfinished business behind. It will only be harder to finish without their help.

Finally, the project leader can't expect to be everyone's friend if you want your project to succeed. Guide your team and your vendors with a firm hand, and throw a completion party to smooth over any bruised egos. And be sure to schedule a wrap-up meeting to discuss what went right and what went wrong. Someday you might have to do this again.