Those of us who have been in the software development game awhile know that inspection of our designs and source code is imperative to reducing defect rates. In four years as a software developer at IBM, I learned firsthand the importance of code inspection early in the life cycle of a project, long before the testing or deployment phases begin.
Unfortunately, even when companies devote adequate time to design, development, and testing, software quality rarely reaches the level required for today's business environment.
Worse, most companies rarely perform thorough inspections due to the intense demands on the workforce and its tendency to be perceived as a bureaucratic barrier that is in conflict with the time to market. Whether this is caused by ever-shortening schedules or increasing pressure from marketing and senior management or both, customers have come to tolerate and even expect software defects. Obviously this situation shouldn't continue any longer than necessary.
A number of vendors offer testing tools capable of assisting developers with tasks such as style adherence and system load generation, but Reasoning, an application service provider (ASP), is putting an interesting twist on software quality testing: service-based code inspection. Reasoning's InstantQA detects critical defects early in the development cycle. As with any code inspection tool or service external to your core development team, InstantQA is not intended to be a replacement for the tools and methodologies that you already use to inspect your code. It does not replace necessary component or system testing, nor does it perform stress or load testing.
Although InstantQA builds on the known value of software inspection as it relates to product quality, customers will likely find a number of other unique advantages: the service provides the neutral advice of a knowledgeable third party, provides an easy-to-read single-source document for defects, and allows staff to remain focused on the core business.
We wanted to evaluate InstantQA using a real, enterprise-class C/C++ application, so we decided to have Reasoning test the open-source code for the latest version of the Apache Web server, which weighs in at just under 60,000 lines of code. To begin the inspection process, customers send the source code to one of Reasoning's inspection centres via a secure FTP link, which provides encryption, or via magnetic media.
Three days after I submitted the code, a 25-page report, in the form of a Microsoft Word document, arrived in my e-mail inbox. The time it takes to receive your report is an issue that customers negotiate with Reasoning before a service contract is signed. The time will vary based on the size and complexity of the application. Also, customers have the choice of receiving their reports in Microsoft Word, Access, or Excel format.
As I expected, the report came with an executive summary, but it was nicely detailed: it stated the project background; the number of lines of code inspected; the number of defects found, which were broken up into categories; and even a defect density ratio, which represents the number of defects per 100,000 lines of code. The report for the Apache code we sent categorised the problems into groups such as memory leaks, out-of-bounds array errors, and null pointer de-references - statements that may access a null pointer.
Following the executive summary were individual detail pages for each error found. The detail pages included a useful top-of-page summary - including type of defect, identification of its location, explanation of the coding anomaly, and the possible negative outcome of the error - along with the offending lines of code printed just below for easy reference.
Getting down to business
InstantQA employs three different processes during its analysis of your application. It combines automated (machine) code inspection technology, defect database analysis, and independent (human) verification by Reasoning staff. By using a combination of three methods on each project, Reasoning greatly increases the likelihood that its service will perform far better than most tools or other quality assurance methods in unearthing hard-to-find defects that will surely decrease the reliability of a software application.
In addition to defect inspection, Reasoning offers two optional services at added cost that check for violations of coding standards or to measure the fragility of software (the likelihood of new defects being introduced during software maintenance).
InstantQA currently supports C, C++, and COBOL. Java support will arrive soon.
Reasoning executives told me that average turnaround is four to five days on an application of approximately 200,000 lines. Larger applications would naturally require a longer turnaround, but as we said earlier, the company negotiates the delivery time on an individual basis. Reasoning officials stated that by year-end they hope to improve on this turnaround time and provide results within 48 hours.
As with any service, InstantQA brings the delicate issue of asset protection to bear. By inspecting for defects at the application level rather than the module level, InstantQA enables up to 100 per cent code path coverage, meaning all parts of the applications are inspected in their entirety. This is wonderful for highlighting crash causes and data-corrupting defects, but it does so at the risk of compromising the security of the application and therefore the customer's competitive advantage. For many organisations, this will not be a risk worth taking. Commodity-type applications such as accounting or order entry should cause little worry. For applications that are more closely tied to company financials and customer advantage, each organisation will have to weigh the benefits and risks carefully before making a decision to employ InstantQA.
Overall, I found InstantQA to have great value, but this nascent company has left more than a few stones unturned: most importantly, the company's service agreement fails to address some essential issues that will concern many customers. For example, there is no wording in the contract that addresses the customer's rights in the event that Reasoning does not meet the agreed-on deadline. In many cases, even a one-day delay could upset a company's entire plan for public release of a product, and Reasoning should provide some way of making itself accountable for such an unforeseen occurrence.
Furthermore, the service agreement that Reasoning uses makes no mention of the company's policy - or the customer's rights - in regard to protection of a customer's copyright or intellectual property rights. Given today's ultra-competitive software market and the desire of most every organisation to protect all intellectual property with hyper-vigilance, we view this as a major drawback.
These oversights - and the lack of Java support - made our evaluation of InstantQA far less enthusiastic than it otherwise might have been. InstantQA's capability of increasing the quality of your software products and business applications, lighten the burden on your development staff, and hasten time to market will make it welcome and overwhelmingly useful to many companies. But until Reasoning works out some of the lingering business issues, we are forced to limit our overall score.the bottom lineInstantQABusin-ess Case: By allowing companies to outsource much of the labour-intensive quality assurance cycle, InstantQA saves money and time over in-house testing in the full life cycle of a software project and increases application quality.
Techn-ology Case: InstantQA provides up to 100 per cent code path coverage and source inspection. It uses a combination of defect databases, automated source verification, and inspection criteria to ensure a complete analysis of C, C++, and COBOL code.
Pros:- -l Quick turnaround time l Requires minimal staff intervention l Cost-effective over hand testingCons-: l Releasing software code to a third party raises security concerns l Java currently not supported l Service agreement lacks essential customer assurancesPlatfor-ms: AnyPrice:- US 10 cents per line of code; volume discounts available. More information available from the company Web site.www.reasoning.com