Liabilities of the Expanding Software Supply Chain

In an increasing number of industries, development organizations are using less of their own code developed in-house and more codes that come from third parties or open-source projects. The extended software supply chain is a business necessity, as go-to-market times are shortened and customer demand for new device features has increased. But this extended supply chain can also turn into a liability if third-party code is not properly tested for defects.

Take the example of the Android kernel that {complink 8103|Coverity Inc.} scanned in November as a part of our annual Scan Open Source Integrity Report. The Android kernel code base is derived from the Linux kernel. {complink 2294|Google} developers then add on their own code and modify some of the existing Linux code and publish the result to mobile OEMs, the ultimate integrators in the mobile supply chain.

Those integrators then take the Android platform from Google and stitch it together with code from third parties that drives devices such as the processor, touchscreen, WiFi, and 3G/4G wireless chips. Then they layer on their own changes to fix integration problems, improve battery life, and provide differentiating features to their phones.

So what happens when a manufacturer or third party finds defects like those that we found during our scan? Working with Google developers, we’ve found that they are happy to fix Android-specific defects, but defects inherited from the Linux kernel are different — the Android team would rather have these defects fixed upstream by Linux kernel maintainers.

This means that they are relying on a community of thousands of open-source software developers to ensure high-quality code and testing, and to respond to issues in a timely fashion. OEMs have similarly wanted to defer defects found in the Android kernel that they did not change, back upstream to Google or the Linux community. We call this the “you-wrote-it-you-fix-it” philosophy of accountability for code quality.

There are many good reasons to want developers who write code to fix it. Developers who wrote the code probably understand it the best. If the defects are fixed in the original source, then when changes are pushed downstream, the integration of those changes will also automatically pick up defect fixes. More subtly, the you-wrote-it-you-fix-it principle also applies to any changes made downstream.

So, for example, if an OEM were to change part of the Android kernel, it could not reasonably expect Google to test those changes or fix defects related to that code. In essence, it would be taking ownership of that code indefinitely, and it would need to bear the cost of integrating future changes to Android with the custom changes it has made. This would potentially slow time to market and add to development and testing costs, so such changes are generally avoided.

This model works well if everyone in the software supply chain has a common standard for quality, but it can be precarious if they don’t. If there were defined metrics for code to meet, then manufacturers could reject the code if the quality is not high enough and hold developers responsible for defects.

But by distributing responsibility among multiple parties without well defined and effective quality control metrics, the result can be uneven quality across different components, even though all of them must work together to provide a high-quality end-user experience. For a given defect, the question of “when will this defect be fixed” cannot be answered unless the third-party and open-source providers are taken into account.

The aerospace industry is an example of a highly regulated area that has been forced to get it right. All avionics software must meet certain coding standards and hold up to rigorous testing. We predict that the automotive industry will set standards of its own in the next few years. Other industries like mobile and independent software vendors need to follow suit to ensure that they can continue to efficiently build products that meet the quality levels their customers demand.

Over time, we expect that as the software supply chain continues to get more complex, some common industry baselines for software integrity will be established without the need for government regulation. We have faith in this idea because it’s a win for everyone: Consumers get better products, and companies get cost savings and faster time to market with higher quality by leveraging their software supply chains more efficiently.

2 comments on “Liabilities of the Expanding Software Supply Chain

  1. prabhakar_deosthali
    February 15, 2011

    In the Software supply chain scenario described in this blog, if the heirarchy is followed  for reporting and fixing the defects, then the software will become manageable during its life-cycle.  At any stage when a problem is detected , the OEM at that stage should first verify whether the bug has got introduced because of the ADD-ONs developed by it. If yes then the bug should be fixed by that OEM itself. Else the bug should be reported to the one-up level in the supply chain heirarchy and that level OEM should fix the bug and pass on the new version or the patch code to the level down.  For any such generic fixes the OEM could notify all other customers also about such problem and the related bug fix . With such mechanism the customer-vendor relationship should remain at the two adjacent levels only.  This means that the OEM at certain level should own  the total liability of all the software it supplies to the customer at the next level but pass on the related liability of the bought out software components  to its upper level.



  2. stochastic excursion
    February 15, 2011

    The developers of code or any other component understand it better in some ways.  In terms of the relevant functionality and performance though, it should be the customer that has the better understanding of the product.  With open source software especially, the cost of ownership is the time and effort required to do just that, take ownership by extensive incoming inspection.  With a product with as many variables as an open sourced operating system kernel, quality assurance requires, on top of the usual evaluation protocols, the involvement of staff-level expertise to ensure all bases are covered.  Open sourced components should also be kept simple, with base functionality to which value can be added using inheritance/aggregation principles.  These object-oriented programming concepts haven't fully been adopted in operating systems, but with applications they prove useful for code maintenance.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.