"In the long run ...

... we're all dead," said the late economist John Maynard Keynes to show his support for short-term government intervention in recessions. But in IT, just what is short term and what is long term? More important, how long is too long to wait for benefits from a project?

"If the return is more than three years out, don't do it, period," advises Nucleus Research's Campbell. His favorite evaluation metric is payback period, and he says three years maps nicely to the life cycle of many technologies.

Payback period is McNulty's favorite metric, too, but the Schwan Food CIO says she's willing to think out further than three years. "If the payback for a project is more than five years, I question it," she says, "and if it's more than 10 years, I won't even consider it."

But Nick Vitalari at BSG Alliance says taking a very long view is sometimes appropriate. He recalls the huge logistics system project more than 18 years ago at Wal-Mart Stores in the US. It resulted in a supply chain so efficient and innovative that, over the ensuing decade, it propelled the company to the top of the retailer heap.

"The investment seemed crazy at the time," he says. "But they took that system and used it as a platform to enable a whole new level of business performance. If they had looked at it just on the basis of ROI or payback period, they wouldn't have seen the long-term portfolio of options that they were creating."

The importance of postmortems

McNulty says the biggest change she has seen in IT cost-benefit analyses in recent years is a new rigor in doing project postmortems and putting lessons learned from them back into the planning process. "In the past, a lot of business cases were put together, but they weren't validated after implementation," she says.

The project postmortems at Schwan Food are done jointly by IT and the business units. As an example of something learned from this process, McNulty says, "When you are defining a project, make sure you have distinct baseline numbers." If multiple simultaneous projects all have improving customer retention as a goal, it won't be clear at the completion of any one project just what led to a change in that customer-retention metric. "So you want to uniquely quantify the benefit and exactly tie it to the initiative," she says.

Far too few companies do such project postmortems, says Christopher Gardner, a co-founder of consulting firm iValue. He says a notable example of a company that does them religiously is The Boeing Company, which has for years "been very careful about learning from mistakes." (In fact, Boeing pioneered applying the concept of learning curves -- in which performance steadily improves as learning occurs -- to its financial and manufacturing processes in the 1930s.)

But, he cautions, if employees know that heads will roll whenever subpar performance is revealed, they will resist postmortems on the projects that need them most.

"It takes discipline and a management open to learning," Gardner concludes.