Liabilities of the Expanding Software Supply Chain

In an increasing number of industries, development organizations are using less of their own code developed in-house and more codes that come from third parties or open-source projects. The extended software supply chain is a business necessity, as go-to-market times are shortened and customer demand for new device features has increased. But this extended supply chain can also turn into a liability if third-party code is not properly tested for defects.

Take the example of the Android kernel that Coverity Inc. scanned in November as a part of our annual Scan Open Source Integrity Report. The Android kernel code base is derived from the Linux kernel. Google (Nasdaq: GOOG) developers then add on their own code and modify some of the existing Linux code and publish the result to mobile OEMs, the ultimate integrators in the mobile supply chain.

Those integrators then take the Android platform from Google and stitch it together with code from third parties that drives devices such as the processor, touchscreen, WiFi, and 3G/4G wireless chips. Then they layer on their own changes to fix integration problems, improve battery life, and provide differentiating features to their phones.

So what happens when a manufacturer or third party finds defects like those that we found during our scan? Working with Google developers, we’ve found that they are happy to fix Android-specific defects, but defects inherited from the Linux kernel are different -- the Android team would rather have these defects fixed upstream by Linux kernel maintainers.

This means that they are relying on a community of thousands of open-source software developers to ensure high-quality code and testing, and to respond to issues in a timely fashion. OEMs have similarly wanted to defer defects found in the Android kernel that they did not change, back upstream to Google or the Linux community. We call this the "you-wrote-it-you-fix-it" philosophy of accountability for code quality.

There are many good reasons to want developers who write code to fix it. Developers who wrote the code probably understand it the best. If the defects are fixed in the original source, then when changes are pushed downstream, the integration of those changes will also automatically pick up defect fixes. More subtly, the you-wrote-it-you-fix-it principle also applies to any changes made downstream.

So, for example, if an OEM were to change part of the Android kernel, it could not reasonably expect Google to test those changes or fix defects related to that code. In essence, it would be taking ownership of that code indefinitely, and it would need to bear the cost of integrating future changes to Android with the custom changes it has made. This would potentially slow time to market and add to development and testing costs, so such changes are generally avoided.

This model works well if everyone in the software supply chain has a common standard for quality, but it can be precarious if they don’t. If there were defined metrics for code to meet, then manufacturers could reject the code if the quality is not high enough and hold developers responsible for defects.

But by distributing responsibility among multiple parties without well defined and effective quality control metrics, the result can be uneven quality across different components, even though all of them must work together to provide a high-quality end-user experience. For a given defect, the question of "when will this defect be fixed" cannot be answered unless the third-party and open-source providers are taken into account.

The aerospace industry is an example of a highly regulated area that has been forced to get it right. All avionics software must meet certain coding standards and hold up to rigorous testing. We predict that the automotive industry will set standards of its own in the next few years. Other industries like mobile and independent software vendors need to follow suit to ensure that they can continue to efficiently build products that meet the quality levels their customers demand.

Over time, we expect that as the software supply chain continues to get more complex, some common industry baselines for software integrity will be established without the need for government regulation. We have faith in this idea because it’s a win for everyone: Consumers get better products, and companies get cost savings and faster time to market with higher quality by leveraging their software supply chains more efficiently.