Defeating the Device Paradigm

Until recently, many people were unfamiliar with the term "defeat device". Following the public disgrace of Volkswagen, it is now believed that a wide range of cars, flat screen TVs and goodness knows what other consumer devices may have been programmed to alter their performance when they detect an official test.

There is a fundamental problem with the regulation of embedded software, which can be captured by a maxim from Arthur Weasley (in Harry Potter and the Chamber of Secrets).

"Never trust anything that can think for itself if you can't see where it keeps its brain." 

As fans of J.K. Rowling will know, Arthur Weasley is a fictional regulator, working for the Misuse of Muggle Artefacts Office at the Ministry of Magic. (Rowling's portrayal of bureaucrats and regulators often verges on the satirical.) Several writers had cited the Weasley maxim in discussions of the Internet of Things even before the Volkswagen story broke - for example Richard Brooker and Kieron O'Hara. As Brooker asks

"How do we make an informed assessment of trust and risk when using a device or service, but with little insight or even awareness of the smart, connected actions that it’s performing in the background, and the information it’s sensing from our interaction and where that information is being sent or used?"

Many people think that regulating the Internet of Things is merely about regulating the devices (the "Things").

But this neglects the importance role of background processes and systems. The Things that make up the Internet of Things are connected to a network, which may be run by a commercial organization for its own commercial purposes.

Take for example a telemonitoring device in a car, which sends location data to a central controller - perhaps controlled by the car company, or an insurance company, or a contractor working for the prison service. This might be part of a car maintenance scheme or a PayAsYouDrive insurance scheme, or a prisoner release system, or perhaps all of the above, but you can't tell this just from inspecting the monitoring device. You have to look at the central controller (the "Brain"), because this is where the signals from the device are processed.

But how should a regulator approach the challenge of testing and monitoring the central processing unit (the "Brain")? Which (for all we know) may be able to detect whether it is being tested, and repurpose the local devices accordingly.

Some writers have argued that Open Source provides the only way to be confident that software is cheat-free. However, two months before the Volkswagen story broke, the US Environment Protection Agency explicitly invoked the Digital Millennium Copyright Act to ban independent experts from testing engine software. As far as I can see, this would prevent the kind of compromise proposed by Kuntal Sampat, which would involve inspecting the interfaces and service calls, as well as full-blown Open Source.

The Internet of Things is not a random collection of devices. It is a safety-critical system of systems, and must be understood (and regulated) as such. But it often suits certain commercial interests to focus our attention on the devices and away from the rest of the system. This is related to what Borgmann calls the Device Paradigm.

---

Ryan Beene, VW emissions ‘defeat device’ isn’t the first (MarketWatch 25 Sept 2015)

Richard Brooker, Can we trust the Internet of (Unsecure) Things? (BT, 7 October 2014) 

Robert Charette, Volkswagen scandal by crooked software (Cutter Consortium, 8 October 2015) (paywall)

Alex Davies, The EPA opposes rules that could've exposed VW's cheating (Wired 18 September 2015)

Terry Fagen, Why Defeat Software Will Never be Used in Medical Devices (Linked-In, 24 September 2015)

James Grimmelmann, Harry Potter and the Mysterious Defeat Device (Slate 22 September 2015)

Russell Hotten, Volkswagen: The scandal explained (BBC News 25 September 2015)

Leo Kelion, VW: Calls to let car software be examined by experts (BBC News 23 September 2015)

Arthur Neslen, Samsung TVs appear less energy efficient in real life than in tests (Guardian 1 October 2015)

Kieron O’Hara, The Fridge’s Brain Sure Ain’t the Icebox (IEEE Internet Computing, Nov/Dec 2014)

Kuntal Sampat, Defeat Device and Open Source (Blogspot 23 September 2015)

Kit Walsh, Researchers Could Have Uncovered Volkswagen’s Emissions Cheat If Not Hindered by the DMCA (Electronic Frontier Foundation, 21 September 2015)

When Code can Kill or Cure (Economist 2 June 2012)

Wikipedia: Defeat Device, Device Paradigm, Life-Critical System

---

Related Posts

Tethering (August 2004)

Understanding the Value Chain of the Internet of Things (June 2015)

The New Economics of Manufacturing (November 2015) 

Uber's Defeat Device and Denial of Service (March 2017)

The Road Less Travelled - Whom Does the Algorithm Serve? (June 2019),

Microsoft becoming boring?

Microsoft is quietly turning itself into a utility company. The vision being presented by many computer industry leaders, including Bill Gates of Microsoft, is of computing services being piped into businesses and homes like electricity, delivered across some kind of grid, and billed according to usage. Utility computing is not just a technological move, but represents a business departure for Microsoft, leading it away from its high-tech alliance with Intel.

Many years ago, Gordon Moore of Intel declared that computer hardware power would double every eighteen months. The hardware industry continues to track this principle – known as Moore’s Law – fairly closely. Meanwhile, the software industry has benefited hugely from Moore’s Law and the exponential growth of computer hardware power. A constant supply of bigger and faster machines has meant a continual demand for new software licences. Meanwhile the software industry has returned the favour by producing feature-laden power-hungry software that helps stimulate the demand for bigger and faster machines.

Moore’s Law appears to offer a vision of ever-cheaper computing – but this is a mirage. Since total production costs, including R&D, are constantly increasing, unit costs can only fall if the total production volumes grow to achieve ever-larger economies of scale. Without rapid growth in IT spending, Moore’s law is not economically viable. Influential computer users, such as Eric Schmidt of Google, are turning their back on Moore’s Law, and using older, cheaper technology. Corporate IT budgets are being squeezed, and purchase of hardware and software is much harder to cost-justify. A dedicated minority of users may want the latest products with the most advanced features, but most users will be focused on more basic aspects of software value, such as reliability and total cost of ownership.

In January 2002 Bill Gates responded to this challenge by declaring a goal of Trustworthy Computing, both for Microsoft and across the industry. Against a background of poor software security, with Microsoft’s own organization frequently falling victim to Internet attack (most recently the Slammer virus), Microsoft promised to improve the reliability and security of its products. Instead of pushing out new features at all costs, Microsoft would devote more effort to testing, and to weeding out vulnerabilities. Some industry observers have seen this initiative as purely a technical one, and have noted Microsoft’s struggle to adopt this consistently across its organization. Others have dismissed it as marketing spin.

But the strategic potential of trustworthy computing goes much further than technical quality. Utility computing requires Microsoft to adopt a different culture, perhaps more grown-up and responsible, less technically exciting. It is a bold strategic move, with no certainty of outcome. Microsoft could simply have chosen to sit through the current downturn in IT spending, waiting for previous spending patterns to be resumed, knowing it would survive where many of its competitors wouldn’t. But if the move is successful, Microsoft becomes a value stock, producing a fat dividend yield from a secure industry position. Safe computing. 

 

Update

In 2010, danah boyd (then and now with Microsoft) wrote a post called Facebook is a utility, utilities get regulated (15 May 2010) HT @j2bryson