Sun's flawed tests delay Java certification

Some Java application server vendors say they have been hampered by hundreds of flaws in test programs that Sun Microsystems provides to make sure these products use Java correctly and work with each other.

Some vendors gripe about wasting weeks struggling with unexpected bugs, while others report only minor problems. The products being tested form the foundation of enterprise Web commerce sites.

Sun engineers patched an unknown number of the almost 6,000 test programs, and 771 went on an "exclusion list," meaning vendors don't have to run these tests to win the coveted "Java-compatible" certification for their application servers.

While the problem has ruffled feathers, the vendors and industry experts say it should not hurt product rollouts.

"[The Java 2 Enterprise Edition specification] is a big animal," says Randy Heffner, an analyst with Giga Information Group. "It's a major standard that can be [interpreted] in different ways. It's normal that vendors will have trouble implementing it."

But the glitches are frustrating for Sun, which released Java 2 Enterprise Edition (J2EE) with great fanfare one year ago. J2EE specifies behaviors, services and APIs as part of a programming model for Java applications on servers.

By all accounts Sun has responded quickly in making fixes to some tests, excluding others, and keeping the test process moving along. Sun engineers are putting the finishing touches on the next version of the J2EE specification, and this time Sun plans a beta test for the new code and the modified programs to test it.

ART Technology Group, BEA Systems, Hitachi and iPlanet are shipping application servers certified as having passed the test suite. Iona Technologies, Silverstream Software and Sybase have announced their products have passed the tests but are not yet shipping.

As vendors began testing their products, they found not only problems with their own code, as intended, but also bugs in the reference implementation and the tests.

"It's a fiasco," says Robert Bickel, executive vice president for Bluestone Software, a Philadelphia application server vendor. "We wasted three weeks on just a couple of issues."

According to Sun, one issue had to do with transactions that involve two or more databases, a feature that is an optional part of the current J2EE Version 1.2 specification. Initially, there were no software drivers available for these distributed transactions, says Karen Tegan, Sun's software engineering manager for J2EE. Bluestone added distributed transactions to its product and then found that Sun's tests could not verify key parts of this type of transaction.