A virtual hit for MLB Advanced Media

I can't resist -- so this is a rebuilding year?

Right. We'll upgrade servers to Solaris 10, upgrade our [storage-area network] infrastructure and replace some older hardware with newer, thinner models that use less power and generate less heat. That data center is in Manhattan, where the cost per square foot is just ridiculous. So, driving up utilization and squeezing everything you can out of every last square inch of rack space is important to us.

We'll move all the services we have running in New York to our data center in Chicago. Migration services is one of the features of virtualization in general, but Solaris Zones specifically. You can do things like clone a zone or migrate a zone. We can move a virtual machine from rack to rack around a single data center, and actually move these services to a virtual machine in a different city.

Also, in addition to seasonal traffic shifts, our load characteristics change drastically during the day. If I have 10 games starting at 7 p.m., there's a huge influx of traffic right at 7 p.m. If we have a bunch of day games, people use their high-speed Internet connections at work, reloading the scoreboard page a lot or watching our flash Gameday product, which has [pitch by pitch updates], or watching the streaming video online. So the ability to slide computing resources around is pretty handy for us.

How else are you using virtualization?

All the services in our new data center will be put into containers, to get the manageability and security benefits -- if there's a security issue, all they've broken into is one virtual machine. Even if a machine has just one service running on it, say one Web server, that's running in a virtualized container. Should the day come when I need to move that service to another piece of hardware, I can just move the virtualized container. My pain-point is really low.

It also lets us accommodate developers who are in a pinch because our season starts this year on March 25 -- the [2007 World Series Champion Boston] Red Sox are opening in Japan against Oakland. That day is hard and fast. Previously, as a security guy, it was my job to say no to developers who wanted to log into a production machine and look at something because they were trying to debug a problem. Especially in the age of [Payment Card Industry] compliance and all that, we need to secure operational access to production machines. But now I can snap off an exact copy of the production machine and hand that to the developer, or I can give him access to a different Solaris Zone running on the same machine. So it let's us draw interesting security lines.

What were the biggest challenges when you were implementing virtualization technology initially?

For every application we run, you end up with some assumptions, such as it will always use this IP address or this much memory. We need to make sure these assumptions are kept to a minimum or at least abstracted out into a different layer or into config files that can be then transformed as part of the virtualized-host boot scripts.

Wrapping our heads around this extra layer of abstraction from an administration perspective is a challenge. If I've got 100 hosts, that's an administration challenge already. If each of those hosts has one or two or three virtual hosts running inside of them, I need to keep track of those as well. And they move around a lot, so you need to be very careful. It seems like we've had to buy three times the number of white boards we use just to keep track of all this stuff.

Right now we're doing most of the management by hand with scripts that we've written ourselves because we've only got, not a toe but maybe most of a foot into the virtualization pool. But we need to get a handle on it before it gets out of control. We're quickly going to outgrow the point where we can manage an army of virtual machines like we can manage a smaller army of hardware because we're doubling our data center capacity on real physical hardware in a couple of months.