Technology Update: Defeating the Device Paradigm

Tuesday, October 13, 2015

Defeating the Device Paradigm

Until recently, many people were unfamiliar with the term "defeat device". Following the public disgrace of Volkswagen, it is now believed that a wide range of cars, flat screen TVs and goodness knows what other consumer devices may have been programmed to alter their performance when they detect an official test.

There is a fundamental problem with the regulation of embedded software, which can be captured by a maxim from Arthur Weasley (in Harry Potter and the Chamber of Secrets).

"Never trust anything that can think for itself if you can't see where it keeps its brain."

As fans of J.K. Rowling will know, Arthur Weasley is a fictional regulator, working for the Misuse of Muggle Artefacts Office at the Ministry of Magic. (Rowling's portrayal of bureaucrats and regulators often verges on the satirical.) Several writers had cited the Weasley maxim in discussions of the Internet of Things even before the Volkswagen story broke - for example Richard Brooker and Kieron O'Hara. As Brooker asks

"How do we make an informed assessment of trust and risk when using a device or service, but with little insight or even awareness of the smart, connected actions that it’s performing in the background, and the information it’s sensing from our interaction and where that information is being sent or used?"

Many people think that regulating the Internet of Things is merely about regulating the devices (the "Things").

But this neglects the importance role of background processes and systems. The Things that make up the Internet of Things are connected to a network, which may be run by a commercial organization for its own commercial purposes.

Take for example a telemonitoring device in a car, which sends location data to a central controller - perhaps controlled by the car company, or an insurance company, or a contractor working for the prison service. This might be part of a car maintenance scheme or a PayAsYouDrive insurance scheme, or a prisoner release system, or perhaps all of the above, but you can't tell this just from inspecting the monitoring device. You have to look at the central controller (the "Brain"), because this is where the signals from the device are processed.

But how should a regulator approach the challenge of testing and monitoring the central processing unit (the "Brain")? Which (for all we know) may be able to detect whether it is being tested, and repurpose the local devices accordingly.

Some writers have argued that Open Source provides the only way to be confident that software is cheat-free. However, two months before the Volkswagen story broke, the US Environment Protection Agency explicitly invoked the Digital Millennium Copyright Act to ban independent experts from testing engine software. As far as I can see, this would prevent the kind of compromise proposed by Kuntal Sampat, which would involve inspecting the interfaces and service calls, as well as full-blown Open Source.

The Internet of Things is not a random collection of devices. It is a safety-critical system of systems, and must be understood (and regulated) as such. But it often suits certain commercial interests to focus our attention on the devices and away from the rest of the system. This is related to what Borgmann calls the Device Paradigm.

Ryan Beene, VW emissions ‘defeat device’ isn’t the first (MarketWatch 25 Sept 2015)

Richard Brooker, Can we trust the Internet of (Unsecure) Things? (BT, 7 October 2014)

Robert Charette, Volkswagen scandal by crooked software (Cutter Consortium, 8 October 2015) (paywall)

Alex Davies, The EPA opposes rules that could've exposed VW's cheating (Wired 18 September 2015)

Terry Fagen, Why Defeat Software Will Never be Used in Medical Devices (Linked-In, 24 September 2015)

James Grimmelmann, Harry Potter and the Mysterious Defeat Device (Slate 22 September 2015)

Russell Hotten, Volkswagen: The scandal explained (BBC News 25 September 2015)

Leo Kelion, VW: Calls to let car software be examined by experts (BBC News 23 September 2015)

Arthur Neslen, Samsung TVs appear less energy efficient in real life than in tests (Guardian 1 October 2015)

Kieron O’Hara, The Fridge’s Brain Sure Ain’t the Icebox (IEEE Internet Computing, Nov/Dec 2014)

Kuntal Sampat, Defeat Device and Open Source (Blogspot 23 September 2015)

Kit Walsh, Researchers Could Have Uncovered Volkswagen’s Emissions Cheat If Not Hindered by the DMCA (Electronic Frontier Foundation, 21 September 2015)

When Code can Kill or Cure (Economist 2 June 2012)

Wikipedia: Defeat Device, Device Paradigm, Life-Critical System