The Linux kernel development process is akin to a game of leapfrog. Even-numbered kernels (v2.0, v2.2, v2.4) are stable kernels, and odd-numbered kernels are unstable, or development, kernels. As soon as a kernel is released as an even number, the next odd numbered kernel is born, and historically, the previous even-numbered kernel is placed into maintenance mode.
As such, with the advent of v2.6, Linux kernel v.2.4 will be maintained with security patches and bug fixes, but official releases of the v2.4 kernel will not contain new features; major updates will be handled in the v2.6 branch. Also following the official release of v2.6, the v2.7 kernel will soon be born and become the rough framework for the as-yet-unspecified v2.8 kernel.
During the development process, the Linux Kernel Mailing List (LKML) is the main sounding board for the kernel developers, and in fact, anyone developing core Linux code. The list is not restricted, so any Linux user may post questions or commentary regarding any Linux kernel. Because any question for the list may have been addressed previously, the LKML archives are a great resource to those working with the internals of the Linux kernel or trying to debug a problem.
During the course of my kernel testing, I ran into a few interesting anomalies with the v2.6-test kernel series and posted my findings and thoughts to the list. The first post detailed an issue regarding poor per-character performance of the v2.6.0-test8 kernel. Less than 30 minutes later, Linux leader Linus Torvalds, had responded to my post and offered some suggestions, followed by Bill Rugolsky Jr. and Andrew Morton adding insight. Within six hours, the problem had been reliably reproduced by a number of core Linux kernel developers, and traced to an issue with a library linked by the Native POSIX Thread Library (NPTL) code. The problem was fixed by Ulrich Drepper of Red Hat and incorporated into the next release.
A few days later, I noticed another performance issue. The Samba tests under the v2.6 kernel were ridiculously slow. The numbers I was seeing on the Intel Xeon system were truly terrible. I had previously tested the lower-level kernel subsystems on this server and seen no problems with performance, but the higher-level tests showed a significant bottleneck somewhere. Another post to the LKML, followed by responses from Torvalds and Morton, helped me identify the source of the problem.
I had originally configured my array controller, LSI Logic’s Fusion-MPT, without RAID so that each disk was viewed as a stand alone SCSI drive. I did the I/O subsystem tests with this configuration and noted very high performance results. During a rebuild of the server, however, I reconfigured the disks into a RAID1 array. I then performed the Samba tests, which produced abysmal performance numbers. Naturally, that didn’t make sense.
The problem was with the Fusion-MPT drivers. With a RAID array built on the controller, the MPT drivers simply don’t function properly, rendering the overall system very sluggish. Reconfiguring the controller to use stand alone drives fixed the problem, and the performance was stellar. This issue was relayed to the developers responsible for the Fusion-MPT drivers, but a fix has yet to be incorporated into the official kernel, and v2.6 was released with this bug intact.
These are perfect examples of how the stable, even-numbered Linux kernel is developed and debugged. Meanwhile, a similar process is taking place with the unstable, odd-numbered kernel. As development occurs on the unstable kernel and continues to occur on the production kernel, some code is shared between the two projects. Sometimes features will be back-ported but only after much debate, and the number is usually minimal.
Support for new architectures, such as Advanced Micro Devices’ Opteron, are generally born as external projects that contribute code to the official kernel. In fact, bleeding-edge hardware support is often available only as a patch to the production kernel; this is why compilations of v2.4 and v2.6 kernels on Opteron require patches to successfully build. Once they’ve proven reliable, these patches are merged into the official production kernel and included in the next official release. Patches may also become available for the unstable kernels and primed for inclusion in the next production release.
Decisions on what features and patches are incorporated into the official kernel are generally preceded by much debate among kernel developers, but are ultimately made by the kernel maintainer, a central authority who shoulders the brunt of day-to-day maintenance, as well as the responsibility for official kernel releases.
Given the size and scope of the kernel, neither the maintainers nor Linus Torvalds himself can fully know and understand every portion of the kernel. To alleviate this, several unofficial kernel subsystem maintainers are entrusted to keep a watchful eye on their chosen sections of the code and to contribute validated patch sets to the maintainer for inclusion in the next release.
New releases of the stable kernel are vetted through a release candidate process, during which kernel patches are tested by the community. In addition to release candidate kernels, patches for stable and unstable kernels are distributed by a select few core developers, such as Alan Cox and Andrew Morton. These patches usually contain experimental code that hasn’t been officially introduced into the source by Torvalds, as well as bug fixes or hardware support likely to be incorporated into the next release.
While the kernel maintainers are responsible for the kernel under their care, Linus Torvalds still runs the show. Officially, Linus is the persistent maintainer of the current development kernel, and he hand-picks the maintainer of the new stable kernel when release time approaches.
Drawing on hundreds of developers, a few maintainers, and a Q&A team in the thousands, the Linux kernel keeps on rolling.