#paxtechnica Some further thoughts arising from the @CRASSHlive conference in Cambridge on The Implications of the Internet of Things. (For a comprehensive account, see @LaurieJ's livenotes.)
Many people are worried about the security implications of the Internet of Things. The world is being swamped with cheap internet-enabled devices. As the manufacturing costs, size and power consumption of these devices are being driven down, most producers have neither the expertise not the capacity to build any kind of security into them.
One of the reasons why this problem is increasing is that it is cheaper to use a general-purpose chip than to design a special purpose chip. So most IoT devices have far more processing power and functionality than they strictly need. This extra functionality can be then coopted for covert or malicious purposes. IoT devices may easily be recruited into a global botnet, and devices from some sources may even have been covertly designed for this purpose.
Sensors are bad enough - baby monitors and sex toys. Additional concerns apply to IoT actuators - devices that can produce physical effects. For example, lightbulbs that can flash (triggering epileptic fits), thermostats that can switch on simultaneously across a city (melting the grid), centrifuges that can spin out of control (attempting to sabotage Iran's nuclear capability).
Jon Crowcroft proposed that some of this could be addressed in terms of safety and liability. Safety is a useful driver for increased regulation, and insurance companies will be looking for ways to protect themselves and their corporate customers. While driverless cars generate much discussion, similar questions of safety and liability arise from any cars containing significant quantities of new technology. What if the brake algorithm fails? And given the recent history of cheat software by car manufacturers, can we trust the car not to alter the driver logs in order to evade liability for an accident?
In many cases, the consumer can be persuaded that there are benefits from internet-enabled devices, and these benefits may depend on some level of interoperability between multiple devices. But we aren't equipped to reason about the trade-off between accessibility/usability and security/privacy.
For comparison's sake, consider a retailer who has to decide whether to place the merchandise in locked glass cases or on open shelves. Open shelves will result in more sales, but also more shoplifting. So the retailer locks up the jewelry but not the pencils or the furniture, and this is based on a common-sense balance of value and risk.
But with the Internet of Things, people generally don't have a good enough understanding of value and risk to be able to reason intelligently about this kind of trade-off. Philip Howard advises users to appreciate that devices "have an immediate function that is useful to you and an indirect function that is useful to others" (p255). But just knowing this is not enough. True security will only arise when we have the kind of transparency (or visibility or unconcealment) that I referenced in my previous post.
Related Posts
Defeating the Device Paradigm (October 2015)
Pax Technica - The Book (November 2017)
Pax Technica - The Conference (November 2017)
The Smell of Data (December 2017)
Outdated Assumptions - Connectivity Hunger (June 2018)
References
Cory Doctorow, The Coming War on General Computation (2011)
Carl Herberger, How hackers will exploit the Internet of Things in 2017 (HelpNet Security, 14 November 2016)
Philip Howard, Pax Technica: How The Internet of Things May Set Us Free or Lock Us Up (Yale 2015)
Laura James, Pax Technica Notes (Session 1, Session 2, Session 3, Session 4)
Holly Robbins, The Path for Transparency for IoT Technologies (ThingsCon, June 2017)
Jack Wallen, Five nightmarish attacks that show the risks of IoT security (ZDNet, 1 June 2017)
Showing posts with label risk-trust-security. Show all posts
Showing posts with label risk-trust-security. Show all posts
Saturday, November 25, 2017
Sunday, November 08, 2015
How Soon Might Humans Be Replaced At Work?
#CIPAai An interesting debate on Artificial Intelligence took place at the Science Museum this week, sponsored by the Chartered Institute of Patent Agents. When will humans be replaced by computers in any given job?
As this was the professional body for Patent Agents, they decided to pick an example close to their hearts. The specific motion being debated was that a patent would be filed and granted without human intervention within the next 25 years. The motion was passed roughly 80-60.
At first sight, this debate appeared to be an exercise in technological forecasting. When would AI be capable of creating new inventions and correctly drafting the patent application? And when would AI be capable of evaluating a patent application, carrying out the necessary searches, and granting a patent. Is this the kind of thing we should expect when the much vaunted Singularity (predicted from around 2040 onwards) occurs?
Speaking for the motion, Calum Chase and Chrissie Lightfoot were enthusiastic about the technological opportunities of AI. They pointed out the incredible feats that were already achieved as a result of machine learning, including some surprisingly creative solutions to technical problems.
Speaking against the motion, Nigel Hanley and Ilya Kazi acknowledged the great contribution of computer intelligence to support the patent agent and patent examiner, but were sceptical that anyone would trust a computer with such an important task as filing and granting patents. Nigel Hanley pointed out the limitations of internet search, which is of course designed to find things that other people have already found. (As A.A. Milne put it, Thinking With The Majority.)
The motion only required that a single patent be filed and granted without human intervention. It didn't need to be a particularly complicated one. But even to grant a single patent without human intervention would require a change in the law, presumably agreed internationally. (As it happens, my late father Kenneth Veryard was involved in the development of European Patent Law around 25 years ago, so I am aware of the time and painstaking effort required to achieve such international agreements.)
But this reframes the debate: from a technological one about the future capability of computers, to a sociopolitical one about the possibility of institutional change. Even if some algorithm were good enough to compete with humans, at least for some routine patent matters, the question is whether politicians would be willing to entrust these matters to an algorithm.
There are also strange questions of ownership and rights. Examples of computer intelligence always seem to come back to the usual suspects - Google, IBM Watson, and their ilk. If the creativity comes from the large computer networks run by these companies, then the patents will belong to these corporations. When Thomas Watson said, "I think there is a world market for maybe five computers", he wasn't talking about billions of laptops or trillions of internet-enabled things, but the very much smaller number of major computer networks capable of controlling everything else.
Can we realistically expect AI to take over one small area of patent law without taking over the much larger challenge of cleaning up legislation? After all, a genuine superintelligence might well come up with a much better basis for promoting innovation and protecting the interests of inventors than a few ancient principles of patent law.
But perhaps here's the killer argument. As the volume of patent applications increases, the cost of processing them all by hand becomes prohibitive. So governments could be tempted by the cost-savings offered by a clever algorithm. Even though governments have a very bad track record at realising cost savings from IT projects, politicians can often be persuaded to think it will be different this time.
So even if AI patent activity turns out not to be as good as when humans do it, and even if it subsequently results in a lot of seriously expensive litigation, it could seem a lot cheaper in the short-term.
As this was the professional body for Patent Agents, they decided to pick an example close to their hearts. The specific motion being debated was that a patent would be filed and granted without human intervention within the next 25 years. The motion was passed roughly 80-60.
At first sight, this debate appeared to be an exercise in technological forecasting. When would AI be capable of creating new inventions and correctly drafting the patent application? And when would AI be capable of evaluating a patent application, carrying out the necessary searches, and granting a patent. Is this the kind of thing we should expect when the much vaunted Singularity (predicted from around 2040 onwards) occurs?
Speaking for the motion, Calum Chase and Chrissie Lightfoot were enthusiastic about the technological opportunities of AI. They pointed out the incredible feats that were already achieved as a result of machine learning, including some surprisingly creative solutions to technical problems.
Speaking against the motion, Nigel Hanley and Ilya Kazi acknowledged the great contribution of computer intelligence to support the patent agent and patent examiner, but were sceptical that anyone would trust a computer with such an important task as filing and granting patents. Nigel Hanley pointed out the limitations of internet search, which is of course designed to find things that other people have already found. (As A.A. Milne put it, Thinking With The Majority.)
The motion only required that a single patent be filed and granted without human intervention. It didn't need to be a particularly complicated one. But even to grant a single patent without human intervention would require a change in the law, presumably agreed internationally. (As it happens, my late father Kenneth Veryard was involved in the development of European Patent Law around 25 years ago, so I am aware of the time and painstaking effort required to achieve such international agreements.)
But this reframes the debate: from a technological one about the future capability of computers, to a sociopolitical one about the possibility of institutional change. Even if some algorithm were good enough to compete with humans, at least for some routine patent matters, the question is whether politicians would be willing to entrust these matters to an algorithm.
There are also strange questions of ownership and rights. Examples of computer intelligence always seem to come back to the usual suspects - Google, IBM Watson, and their ilk. If the creativity comes from the large computer networks run by these companies, then the patents will belong to these corporations. When Thomas Watson said, "I think there is a world market for maybe five computers", he wasn't talking about billions of laptops or trillions of internet-enabled things, but the very much smaller number of major computer networks capable of controlling everything else.
Can we realistically expect AI to take over one small area of patent law without taking over the much larger challenge of cleaning up legislation? After all, a genuine superintelligence might well come up with a much better basis for promoting innovation and protecting the interests of inventors than a few ancient principles of patent law.
But perhaps here's the killer argument. As the volume of patent applications increases, the cost of processing them all by hand becomes prohibitive. So governments could be tempted by the cost-savings offered by a clever algorithm. Even though governments have a very bad track record at realising cost savings from IT projects, politicians can often be persuaded to think it will be different this time.
So even if AI patent activity turns out not to be as good as when humans do it, and even if it subsequently results in a lot of seriously expensive litigation, it could seem a lot cheaper in the short-term.
References
http://www.cipadebate.org.uk/
Steven Johnson, Superintelligence Now (How We Get To Next, 28 October 2015)
James Nurton, Could a computer do your job (Managing IP, 3 November 2015)
Wikipedia: Technological Singularity
Related Posts
The End of Google (June 2006), What does a patent say? (February 2023)
Update 2016
For the potential ramifications of robotic legal assistants, see Remus, Dana and Levy, Frank S., Can Robots Be Lawyers? Computers,
Lawyers, and the Practice of Law (December 30, 2015). Available at
SSRN: http://ssrn.com/abstract=2701092 or http://dx.doi.org/10.2139/ssrn.2701092. Reported by Aviva Rutkin, Artificial intelligence could make lawyers more risk averse (New Scientist 27 January 2016).
See also Ryan Abbott, I Think, Therefore I Invent: Creative Computers and the Future of Patent Law (Boston College Law Review, Vol 57 Issue 4, September 2016). Reported in Iain Thompson, AI software should be able to register its own patents, law prof argues (The Register, 17 October 2016)
Update 2021
Tom Knowles, Patently brilliant ... AI listed as inventor for first time (The Times, 28 July 2021)
Dagmar Monett tweeted Can an #AI invent something? No, it can't.
David Gunkel replied I understand the issue here, but the question before the court in "Thaler v Commissioner of Patents [2021] FCA 879" was not "Can an #AI invent something?" The question decided by the court was "Can an #AI (DABUS) be named "inventor" on a patent application?" Different questions.
Update 2023
Further news on the DABUS case
AI cannot patent inventions, UK Supreme Court confirms (BBC News 21 December 2023)
Update 2025
How soon might humans be replaced at work (July 2025)
updated 18 October 2021, link added 21 Feb 2023, updated 22 Dec 2023
Thursday, November 27, 2014
Misunderstanding CRM and Big Data
Listening to @peter_w_ryan, @markhillary and Alexey Minkevich talking about #CRM and #BigData at the Institute of Directors, sponsored by IBA Group.
Peter cites an Ovum survey showing that Customer Satisfaction is now the number one concern of management, and argues for what Ovum calls Intelligent CRM. (CA announced something under this label back in October 2000. Other products are available.)
Mark says that CRM and Big Data are widely misunderstood, which is certainly true. My own opinion is the first misunderstanding is to think CRM is about managing THE relationship with THE customer, and I completely agree with Clayton Christensen (via Sloan) that this isn't enough. What we really need to focus on is the job the customers are trying to get done when they use your product or service.
Who is good at CRM? Peter cites an example of a professor of marketing who got a personalized service at a certain chain of hotels and has been talking about it ever since. (That's a pretty good coup for the hotel, if we take the story at face value.) Mark cites the video game market, where both the console manufacturers and the large game publishers are able to collect and analyse huge quantities of consumer behaviour.
Is CRM with Big Data merely a new way of taking advantage of customers? Although most people seem oblivious to the privacy and trust risks, the Wall Street Journal this week suggested that the consumer is becoming more savvy and less susceptible to exploitative loyalty schemes and promotions. This might help to explain why Tesco, once a master of the science of retail, now seems to be faltering.
If there is a sustainable business model based on CRM and Big Data, it must surely involve using these technologies to engage intelligently, authentically and ethically with customers, rather than imagining that these technologies can provide a quick fix for stupid organizations to take advantage of compliant customers.
Related Blogs
Customer Orientation (May 2009)
The Science of Retail (April 2012)
Other Articles
Martha Mangelsdorf, Understanding your customer isn't enough (Sloan Review May 2009)
Shelly Banjo and Sara Germano, The End of the Impulse Shopper (Wall Street Journal 25 November 2014)
Intelligent CRM
AI-CRM "An intelligent CRM system with atuo-learning-tunning engine (sic), Aichain offers the most widely used open source business intelligence software in the world." Last updated March 2013
CA rolling out customer relationship management software (ComputerWorld October 2000)
IBA Group "maintains its focus on IT outsourcing that has become a strategy for many organizations seeking to improve their business processes"
Peter cites an Ovum survey showing that Customer Satisfaction is now the number one concern of management, and argues for what Ovum calls Intelligent CRM. (CA announced something under this label back in October 2000. Other products are available.)
Mark says that CRM and Big Data are widely misunderstood, which is certainly true. My own opinion is the first misunderstanding is to think CRM is about managing THE relationship with THE customer, and I completely agree with Clayton Christensen (via Sloan) that this isn't enough. What we really need to focus on is the job the customers are trying to get done when they use your product or service.
Who is good at CRM? Peter cites an example of a professor of marketing who got a personalized service at a certain chain of hotels and has been talking about it ever since. (That's a pretty good coup for the hotel, if we take the story at face value.) Mark cites the video game market, where both the console manufacturers and the large game publishers are able to collect and analyse huge quantities of consumer behaviour.
Is CRM with Big Data merely a new way of taking advantage of customers? Although most people seem oblivious to the privacy and trust risks, the Wall Street Journal this week suggested that the consumer is becoming more savvy and less susceptible to exploitative loyalty schemes and promotions. This might help to explain why Tesco, once a master of the science of retail, now seems to be faltering.
If there is a sustainable business model based on CRM and Big Data, it must surely involve using these technologies to engage intelligently, authentically and ethically with customers, rather than imagining that these technologies can provide a quick fix for stupid organizations to take advantage of compliant customers.
Related Blogs
Customer Orientation (May 2009)
The Science of Retail (April 2012)
Other Articles
Martha Mangelsdorf, Understanding your customer isn't enough (Sloan Review May 2009)
Shelly Banjo and Sara Germano, The End of the Impulse Shopper (Wall Street Journal 25 November 2014)
Intelligent CRM
AI-CRM "An intelligent CRM system with atuo-learning-tunning engine (sic), Aichain offers the most widely used open source business intelligence software in the world." Last updated March 2013
CA rolling out customer relationship management software (ComputerWorld October 2000)
IBA Group "maintains its focus on IT outsourcing that has become a strategy for many organizations seeking to improve their business processes"
Thursday, January 17, 2013
Business Signal Optimization
@DouglasMerrill of @ZestFinance (via @dhinchcliffe) tells us A Practical Approach to Reading Signals in Data (HBR Blogs November 2012)
If we think of data in tabular form, there are two obvious ways of increasing the size of the table - increasing the number of rows (greater volume of cases) or increasing the number of columns (greater volume of signals). This can either involve a greater variety of variables, as Merrill advocates, or a higher frequency of the same variable. I have talked in the past about the impact of increased granularity on Big Data.
As I understand it, Merrill's company sells Big Data solutions to the insurance underwriting industry, and its algorithms use thousands of different indicators to calculate risk.
The first question I always have in regard to such sophisticated decision-support technologies is what the feedback and monitoring loop looks like. If the decision is fully automated, then it would be good to have some mechanism to monitor the accuracy of the algorithm's predictions. Difficulty here is that there is usually no experimental control, so there is no direct way of learning whether the algorithm is being over-cautious. I call this one-sided learning,
Where the decision involves some human intervention, this gives us some further things to think about in evaluating the effectiveness of the decision-support. What are the statistical patterns of human intervention, and how do these relate to the way the decision-support software presents its recommendations?
Suppose that statistical analysis shows that the humans are basing their decisions on a much smaller subset of indicators, and that much of the data being presented to the human decision-makers is being systematically ignored. This could mean either that the software is too complicated (over-engineered) or that the humans are too simple-minded (under-trained). I have asked many CIOs whether they carry out this kind of statistical analysis, but most of them seem to think their responsibility for information management ends when they have provided the users with the requested information or service, therefore how this information or service is used is not their problem.
Meanwhile, the users may well have alternative sources of information, such as social media. One of the challenges Dion Hinchcliffe raises is how these richer sources of information can be integrated with the tabular data on which the traditional decision-support tools are based. I think this is what Dion means by "closing the clue gap".
Dion Hinchcliffe, The enterprise opportunity of Big Data: Closing the "clue gap" (ZDNet August 2011)
Dion Hinchcliffe, How social data is changing the way we do business (ZDNet Nov 2012)
Douglas Merrill, A Practical Approach to Reading Signals in Data (HBR Blogs November 2012)
Places are still available on my forthcoming workshops Business Awareness (Jan 28), Business Architecture (Jan 29-31), Organizational Intelligence (Feb 1).
If we think of data in tabular form, there are two obvious ways of increasing the size of the table - increasing the number of rows (greater volume of cases) or increasing the number of columns (greater volume of signals). This can either involve a greater variety of variables, as Merrill advocates, or a higher frequency of the same variable. I have talked in the past about the impact of increased granularity on Big Data.
As I understand it, Merrill's company sells Big Data solutions to the insurance underwriting industry, and its algorithms use thousands of different indicators to calculate risk.
The first question I always have in regard to such sophisticated decision-support technologies is what the feedback and monitoring loop looks like. If the decision is fully automated, then it would be good to have some mechanism to monitor the accuracy of the algorithm's predictions. Difficulty here is that there is usually no experimental control, so there is no direct way of learning whether the algorithm is being over-cautious. I call this one-sided learning,
Where the decision involves some human intervention, this gives us some further things to think about in evaluating the effectiveness of the decision-support. What are the statistical patterns of human intervention, and how do these relate to the way the decision-support software presents its recommendations?
Suppose that statistical analysis shows that the humans are basing their decisions on a much smaller subset of indicators, and that much of the data being presented to the human decision-makers is being systematically ignored. This could mean either that the software is too complicated (over-engineered) or that the humans are too simple-minded (under-trained). I have asked many CIOs whether they carry out this kind of statistical analysis, but most of them seem to think their responsibility for information management ends when they have provided the users with the requested information or service, therefore how this information or service is used is not their problem.
Meanwhile, the users may well have alternative sources of information, such as social media. One of the challenges Dion Hinchcliffe raises is how these richer sources of information can be integrated with the tabular data on which the traditional decision-support tools are based. I think this is what Dion means by "closing the clue gap".
Dion Hinchcliffe, The enterprise opportunity of Big Data: Closing the "clue gap" (ZDNet August 2011)
Dion Hinchcliffe, How social data is changing the way we do business (ZDNet Nov 2012)
Douglas Merrill, A Practical Approach to Reading Signals in Data (HBR Blogs November 2012)
Places are still available on my forthcoming workshops Business Awareness (Jan 28), Business Architecture (Jan 29-31), Organizational Intelligence (Feb 1).
Thursday, September 01, 2011
Black Swan Blindness
In my post Black Swans and Complex System Failure, I talked about the architectural implications of some recent disasters, including the Gulf of Mexico oil spillage in 2010 and the partial melt-down in Japanese nuclear reactors following the tsunami in 2011. Both of these disasters involved something that isn't supposed to happen: the simultaneous failure of multiple fail-safe mechanisms.
A new study by Oxford University and McKinsey finds a similar phenomenon in technology investment, where large IT projects may experience spiralling costs as a result of multiple problems occurring simultaneously. According to the researchers, this is up to twenty times more frequent than traditional risk modelling techniques would expect, with one in six large IT projects going over budget by an average of over 200%. Researchers refer to the tendency to disregard rare but high-impact problems/risks as black swan blindness.
As an example, Professor Bent Flyvbjerg cites the collapse of Auto Windscreens, which went into administration in February following a disastrous attempt to implement a new IT system. "Black swans often start as purely software issues. But then several things can happen at the same time - economic downturn, financial difficulties - which compound the risk," he explained.
Professor Flyvbjerg has coined the term Black Swan Management, which currently merits its own Wikipedia page. Simon Moore (author of Strategic Project Portfolio Management) questions whether it is appropriate to use the term "black swan" for something that occurs with a one in six probability, but supports Flyvbjerg's conclusion that when projects go wrong they can go extremely wrong.
Flyvbjerg makes five fairly bland recommendations for avoiding IT project failure, including recruiting a "master builder". Some people may interpret this as an endorsement of the large IT service firms, but these firms have been responsible for some of the most extravagent failures. Is there any evidence that master builders are any more immune from "black swan blindness" than anyone else? Indeed, as a Scandinavian, Flyvbjerg will hardly need reminding of Ibsen's portrayal of madness in the play of the same name.
'Black swans' busting IT budgets (BBC News, 26 August 2011)
Bent Flyvbjerg and Alexander Budzier, Why Your IT Project May Be Riskier than You Think (Harvard Business Review, September 2011, pp. 601-603)
Natasha Lomas, Five ways to stop your IT projects spiralling out of control and overbudget (Silicon.com, 22 August 2011) (pdf)
Brenda Michelson, Complexity, Outliers and the Truth on IT Project Failure (HP Input-Output, 31 Aug 2011)
Simon Moore, Black Swans In Project Management (August 25, 2011)
A new study by Oxford University and McKinsey finds a similar phenomenon in technology investment, where large IT projects may experience spiralling costs as a result of multiple problems occurring simultaneously. According to the researchers, this is up to twenty times more frequent than traditional risk modelling techniques would expect, with one in six large IT projects going over budget by an average of over 200%. Researchers refer to the tendency to disregard rare but high-impact problems/risks as black swan blindness.
As an example, Professor Bent Flyvbjerg cites the collapse of Auto Windscreens, which went into administration in February following a disastrous attempt to implement a new IT system. "Black swans often start as purely software issues. But then several things can happen at the same time - economic downturn, financial difficulties - which compound the risk," he explained.
Professor Flyvbjerg has coined the term Black Swan Management, which currently merits its own Wikipedia page. Simon Moore (author of Strategic Project Portfolio Management) questions whether it is appropriate to use the term "black swan" for something that occurs with a one in six probability, but supports Flyvbjerg's conclusion that when projects go wrong they can go extremely wrong.
Flyvbjerg makes five fairly bland recommendations for avoiding IT project failure, including recruiting a "master builder". Some people may interpret this as an endorsement of the large IT service firms, but these firms have been responsible for some of the most extravagent failures. Is there any evidence that master builders are any more immune from "black swan blindness" than anyone else? Indeed, as a Scandinavian, Flyvbjerg will hardly need reminding of Ibsen's portrayal of madness in the play of the same name.
'Black swans' busting IT budgets (BBC News, 26 August 2011)
Bent Flyvbjerg and Alexander Budzier, Why Your IT Project May Be Riskier than You Think (Harvard Business Review, September 2011, pp. 601-603)
Natasha Lomas, Five ways to stop your IT projects spiralling out of control and overbudget (Silicon.com, 22 August 2011) (pdf)
Brenda Michelson, Complexity, Outliers and the Truth on IT Project Failure (HP Input-Output, 31 Aug 2011)
Simon Moore, Black Swans In Project Management (August 25, 2011)
Wednesday, June 08, 2011
Ethics of Risk in Public Sector IT
@tonyrcollins via @glynmoody and @Mark_Antony asks Should winning bidders tell if they suspect a new contract is undeliverable? (8 June 2011) and raises some excellent ethical points about public sector procurement.
One of the functions of good journalism is to hold people and organizations to account. Tony fishes out a speech given in 2004 by Sir Christopher Bland, then chairman of BT, in which he acknowledged incomplete success in previous ventures, and admitted the extraordinary challenges involved in the NPfIT, for which BT had just won three contracts then valued at over £2bn.
There is obviously a difference between something's being extremely difficult and its being impossible. BT executives can fairly claim that they were always open about the chance that it was going to be difficult, and that they didn't know for sure that it was going to be impossible. But at the same time, there is an asymmetry of information here - the supplier is presumably in a better position to assess certain classes of risk than the customer. (Meanwhile, there may be other classes of risk that the customer should know more about than the supplier.)
In my opinion, the ethical issues here are not to do with deliberate concealment of known facts, but of misleading or inadequate assessment of shared risk. The key word in Tony's headline is the word "suspect". So what are the ethics of doubt?
One of the functions of good journalism is to hold people and organizations to account. Tony fishes out a speech given in 2004 by Sir Christopher Bland, then chairman of BT, in which he acknowledged incomplete success in previous ventures, and admitted the extraordinary challenges involved in the NPfIT, for which BT had just won three contracts then valued at over £2bn.
There is obviously a difference between something's being extremely difficult and its being impossible. BT executives can fairly claim that they were always open about the chance that it was going to be difficult, and that they didn't know for sure that it was going to be impossible. But at the same time, there is an asymmetry of information here - the supplier is presumably in a better position to assess certain classes of risk than the customer. (Meanwhile, there may be other classes of risk that the customer should know more about than the supplier.)
In my opinion, the ethical issues here are not to do with deliberate concealment of known facts, but of misleading or inadequate assessment of shared risk. The key word in Tony's headline is the word "suspect". So what are the ethics of doubt?
Friday, March 04, 2011
IT analysis and trust
@mkrigsman asks "Trust is the currency that matters most. How many analysts / bloggers deserve it?"
@markhillary replies "surely in the same way as a journalist is trusted, by earning it"
@mkrigsman is particularly concerned about those who write about IT failure. (I'm not sure why he singles out that topic, but I note that the concern arose during a conversation with @benioff, boss of Salesforce.) "When someone writes on IT failures ask "What's their angle?". Usually sensationalism, currying favor, or threatening a vendor." When challenged about his own angle by @njames, @mkrigsman replies "I want to expose *why* projects fail, so we understand magnitude of the problem and can improve."
Trust is clearly a difficult issue for software industry analysts. Unfortunately, Michael's answer to Nigel's challenge cannot prove that he doesn't have a hidden agenda, because the untrustworthy are often just as able as the trustworthy to produce a plausible cover story. If we trust Michael it's not because he can answer the challenge but because of his track record.
We also need to ask - trusted by whom. Software companies might prefer industry analysts to be compliant and predictable, but intelligent software users might regard such analysts as being insufficiently independent. Who would you trust to tell you about Microsoft's new platform - someone who is always pro-Microsoft, someone who is always anti-Microsoft, or someone who has a track record of making both positive and negative comments about Microsoft and its competitors?
Of course, this comment doesn't only apply to industry analysts. Robert Scoble, when he worked for Microsoft, made a point of distancing himself from the party line, and he therefore commanded a different kind of attention and respect than did Bill Gates or Steve Ballmer.
From a simplistic software industry perspective, an analyst who talks about IT success might be regarded as a friend, whereas an analyst who talks about IT failure is potentially an enemy. (This might explain Marc Benioff's wish to challenge the hidden agenda of the latter.) While many software and service companies might adopt the from-failure-to-success rhetoric - "the best way to avoid the risk of failure is to buy our software and hire our consultants" - this is not ideal from a sales perspective.
Mark Hillary appeals to a journalistic ethic, which would presumably include things like balance and transparency. But balance is not always appreciated by those with most at stake. In the past, I have written technology reports on new products, which I regarded as generally positive with a few small caveats. (I don't generally waste my time writing about products that are no good.) But the vendors concerned have often regarded my remarks as highly critical. (Fortunately, this over-sensitivity on the part of software companies is now changing, thanks in part to social media, and companies now understand that a robust debate can be just as beneficial as a highly controlled one-way marketing exercise.)
From a narrow software industry perspective, a trustworthy industry analyst is one who satisfies Simon Cameron's definition of an honest politician - "one who, when he is bought, will stay bought". But from a broader perspective, we should surely prefer to trust those industry analysts with independently critical mind, unafraid to ask awkward questions and publish the answers.
With the large industry analysis firms, the question of trust shifts from personal integrity to corporate integrity. The sales pitch for these firms depends not just on isolated flashes of insight from individual analysts, but on the collaborative intelligence of a community of analysts. Corporate integrity depends not just on transparency about the relationship between the work paid for by software vendors and the independent research consumed by CIOs, but also on a coherent and robust research methodology adopted consistently across the firm, typically supported by an apparatus of surveys and structured questionnaires and checklists and spreadsheets. However, there is a potential disconnect between the routine processing of supposedly objective raw data (this product with this market share in this geography in this time period) and the generation of useful interpretation and opinion, which is where the analytical magic and subjectivity comes in. One example of this magic, Gartner's Magic Quadrant, has been challenged in the courts; Gartner's defence has been that MQ represented opinion rather than fact. (See my post The Magic Sorting Hat is Innocent, Okay?) And the complicated relationship between fact and opinion, and the transparency of reasoning and evidence, is surely relevant to the level of trust that can be invested by different stakeholders in such analyses.
By the way, why am I writing about software industry analysis? Obviously, because I want to expose *why* analysis fails, so we understand magnitude of the problem and can improve. How can software industry analysis deliver greater levels of intelligence and value to the software industry as a whole?
@markhillary replies "surely in the same way as a journalist is trusted, by earning it"
@mkrigsman is particularly concerned about those who write about IT failure. (I'm not sure why he singles out that topic, but I note that the concern arose during a conversation with @benioff, boss of Salesforce.) "When someone writes on IT failures ask "What's their angle?". Usually sensationalism, currying favor, or threatening a vendor." When challenged about his own angle by @njames, @mkrigsman replies "I want to expose *why* projects fail, so we understand magnitude of the problem and can improve."
Trust is clearly a difficult issue for software industry analysts. Unfortunately, Michael's answer to Nigel's challenge cannot prove that he doesn't have a hidden agenda, because the untrustworthy are often just as able as the trustworthy to produce a plausible cover story. If we trust Michael it's not because he can answer the challenge but because of his track record.
We also need to ask - trusted by whom. Software companies might prefer industry analysts to be compliant and predictable, but intelligent software users might regard such analysts as being insufficiently independent. Who would you trust to tell you about Microsoft's new platform - someone who is always pro-Microsoft, someone who is always anti-Microsoft, or someone who has a track record of making both positive and negative comments about Microsoft and its competitors?
Of course, this comment doesn't only apply to industry analysts. Robert Scoble, when he worked for Microsoft, made a point of distancing himself from the party line, and he therefore commanded a different kind of attention and respect than did Bill Gates or Steve Ballmer.
From a simplistic software industry perspective, an analyst who talks about IT success might be regarded as a friend, whereas an analyst who talks about IT failure is potentially an enemy. (This might explain Marc Benioff's wish to challenge the hidden agenda of the latter.) While many software and service companies might adopt the from-failure-to-success rhetoric - "the best way to avoid the risk of failure is to buy our software and hire our consultants" - this is not ideal from a sales perspective.
Mark Hillary appeals to a journalistic ethic, which would presumably include things like balance and transparency. But balance is not always appreciated by those with most at stake. In the past, I have written technology reports on new products, which I regarded as generally positive with a few small caveats. (I don't generally waste my time writing about products that are no good.) But the vendors concerned have often regarded my remarks as highly critical. (Fortunately, this over-sensitivity on the part of software companies is now changing, thanks in part to social media, and companies now understand that a robust debate can be just as beneficial as a highly controlled one-way marketing exercise.)
From a narrow software industry perspective, a trustworthy industry analyst is one who satisfies Simon Cameron's definition of an honest politician - "one who, when he is bought, will stay bought". But from a broader perspective, we should surely prefer to trust those industry analysts with independently critical mind, unafraid to ask awkward questions and publish the answers.
With the large industry analysis firms, the question of trust shifts from personal integrity to corporate integrity. The sales pitch for these firms depends not just on isolated flashes of insight from individual analysts, but on the collaborative intelligence of a community of analysts. Corporate integrity depends not just on transparency about the relationship between the work paid for by software vendors and the independent research consumed by CIOs, but also on a coherent and robust research methodology adopted consistently across the firm, typically supported by an apparatus of surveys and structured questionnaires and checklists and spreadsheets. However, there is a potential disconnect between the routine processing of supposedly objective raw data (this product with this market share in this geography in this time period) and the generation of useful interpretation and opinion, which is where the analytical magic and subjectivity comes in. One example of this magic, Gartner's Magic Quadrant, has been challenged in the courts; Gartner's defence has been that MQ represented opinion rather than fact. (See my post The Magic Sorting Hat is Innocent, Okay?) And the complicated relationship between fact and opinion, and the transparency of reasoning and evidence, is surely relevant to the level of trust that can be invested by different stakeholders in such analyses.
By the way, why am I writing about software industry analysis? Obviously, because I want to expose *why* analysis fails, so we understand magnitude of the problem and can improve. How can software industry analysis deliver greater levels of intelligence and value to the software industry as a whole?
Friday, February 18, 2011
Jeopardy and Risk
@Forrester's Andras Cser notes the victory of IBM's Watson computer in a TV quiz game, and asks How Can You Capitalize On This In Risk And Fraud Management?
In his short blogpost, Cser doesn't offer an answer to this question. He merely makes one assertion and one prediction.
Firstly he asserts an easy and superficial connection between the game of Jeopardy and the profession of security, based on "the complexity, amount of unstructured background information, and the real-time need to make decisions." Based on this connection, he makes a bold prediction on behalf of Forrester.
As this is presented as a corporate prediction rather than merely a personal opinion, I'm assuming that this has gone through some kind of internal peer review, and is based on an analytical reasoning process supported by detailed discussions with the IBM team responsible for Watson. I'm assuming Forrester has a robust model of decision-making that justifies Cser's confidence that the Jeopardy victory can be easily translated into the fraud management and data protection domain within the current generation of technology. (Note that the prediction refers to what Watson will be able to do, not what some future computer might be able to do.)
For my part, I have not yet had the opportunity to talk with the IBM team and congratulate them on their victory, but there are some important questions to explore. I think one of the most interesting elements of the Watson victory is not the complexity - which other commentators such as Paul Miller of Engadget have downplayed - but the apparent ability to outwit the other competitors. This ability may well be relevant to a more agile and intelligent approach to security, but that's a long way from the simplistic connection identified by Cser. Meanwhile, I look forward to seeing the evidence that Watson is capable of analysing root causes, which would be a lot harder than winning at Jeopardy.
Paul Miller, Watson wins it all, humans still can do some other cool things (Engadget 16 Feb 2011)
IBM's Watson supercomputer crowned Jeopardy king (BBC News 17 Feb 2011)
In his short blogpost, Cser doesn't offer an answer to this question. He merely makes one assertion and one prediction.
Firstly he asserts an easy and superficial connection between the game of Jeopardy and the profession of security, based on "the complexity, amount of unstructured background information, and the real-time need to make decisions." Based on this connection, he makes a bold prediction on behalf of Forrester.
"Forrester predicts that the same levels of Watson's sophistication will appear in pattern recognition in fraud management and data protection. If Watson can answer a Jeopardy riddle in real time, it will certainly be able to find patterns of data loss, clustering security incidents, and events, and find root causes of them. Mitigation and/or removal of those root causes will be easy, compared to identifying them."
As this is presented as a corporate prediction rather than merely a personal opinion, I'm assuming that this has gone through some kind of internal peer review, and is based on an analytical reasoning process supported by detailed discussions with the IBM team responsible for Watson. I'm assuming Forrester has a robust model of decision-making that justifies Cser's confidence that the Jeopardy victory can be easily translated into the fraud management and data protection domain within the current generation of technology. (Note that the prediction refers to what Watson will be able to do, not what some future computer might be able to do.)
For my part, I have not yet had the opportunity to talk with the IBM team and congratulate them on their victory, but there are some important questions to explore. I think one of the most interesting elements of the Watson victory is not the complexity - which other commentators such as Paul Miller of Engadget have downplayed - but the apparent ability to outwit the other competitors. This ability may well be relevant to a more agile and intelligent approach to security, but that's a long way from the simplistic connection identified by Cser. Meanwhile, I look forward to seeing the evidence that Watson is capable of analysing root causes, which would be a lot harder than winning at Jeopardy.
Paul Miller, Watson wins it all, humans still can do some other cool things (Engadget 16 Feb 2011)
IBM's Watson supercomputer crowned Jeopardy king (BBC News 17 Feb 2011)
Wednesday, April 14, 2010
Enterprise 2.0 inside the firewall?
@infovark 's Dean blogs why he thinks Enterprise 2.0 will fail, and claims that the case for E2.0 inside the firewall is considerably more difficult.
I think the main problem with the case for “E2.0 inside the firewall” is the word “firewall”, which represents an outdated but still common attitude towards maintaining organizational boundaries. I wouldn’t be at all surprised if an organization that relies on firewalls struggles to get the benefits from open distributed business and technology, including Enterprise 2.0.
Dean replies
But the shift away from firewall (sometimes called Deperimeterization) doesn't necessarily entail the second shift Dean mentions, from a 'network of networks' paradigm to a 'unified network' approach, and I am not advocating this. There will perhaps always be limits to interoperability, and there will always be some structure to the network of networks, but this structure will be more open and innovative, and not driven primarily by an obsolete security architecture.
I think the main problem with the case for “E2.0 inside the firewall” is the word “firewall”, which represents an outdated but still common attitude towards maintaining organizational boundaries. I wouldn’t be at all surprised if an organization that relies on firewalls struggles to get the benefits from open distributed business and technology, including Enterprise 2.0.
Dean replies
"It’s true that many forward-thinking organizations are becoming more transparent, and the borders between them are becoming less distinct. Still, eliminating the firewall altogether would require a lot of infrastructure changes. ... An even bigger challenge is the political one. Changing the Internet from a 'network of networks' paradigm to a 'unified network' approach would require far more coordination than most companies — and countries — would be willing to undertake."I agree that shifting away from firewall-based security is a significant strategic move for an organization, not just infrastructure but also political. There are some political issues that would have to be tackled, if the organization is to achieve any potential benefits from Enterprise 2.0.
But the shift away from firewall (sometimes called Deperimeterization) doesn't necessarily entail the second shift Dean mentions, from a 'network of networks' paradigm to a 'unified network' approach, and I am not advocating this. There will perhaps always be limits to interoperability, and there will always be some structure to the network of networks, but this structure will be more open and innovative, and not driven primarily by an obsolete security architecture.
Thursday, January 07, 2010
OWASP Top Ten 2010
@johnccr asks me to give a look to the new OWASP Top Ten 2010 RC1 (pdf), saying "it would be interesting to know if it changed your perception". So here are a few quick comments.
I'm certainly happy to acknowledge that this version makes the limitations of the Top Ten approach much clearer than previous versions, and explicitly encourages organizations to "think beyond the ten risks here". The document is careful not to claim the Top Ten as a full application security program, and warns readers not to stop at ten, because "there are hundreds of issues that could affect the overall security of a web application". But then surely this implies we shouldn't be wasting time reading this document at all; we should be reading the OWASP Developer’s Guide, "which is essential reading for anyone developing web applications today".
The status of the top ten items as risks (rather than, say, weaknesses or vulnerabilities or threats) is also a bit clearer, and the ranking of risks is based on the scale of the risk, not just the frequency of the attack. However, the document also refers to "relatively simple security problems like those in the OWASP Top 10" - which makes it seem that they may be the most obvious rather than the most problematic. Making people aware of simple problems doesn't necessarily promote awareness of more complex problems.
To my mind, the trouble with this kind of list is that it encourages bad thinking. Not only are some risks regarded as more attention-worthy than others (based on a generalized model of risk that may not be relevant to your organization or application portfolio), but each risk is considered in isolation. But a holistic understanding of security and risk needs to look at the composition of risk - how can several apparently small risks sometimes be multiplied into a very large risk.
I'm also concerned about limiting the analysis of risks to application security itself. Presumably a full security risk analysis would need to look at social attacks as well as technical attacks, but the Top Ten are all drawn from the technical side. I looked for this technical focus to be stated and explained somewhere, perhaps in a statement of scope, but couldn't find anything to this effect.
By the way, when I have raised issues about OWASP in the past, I have been challenged to fix them myself. But I'm not a normal member of OWASP, I'm an independent industry analyst who has been asked by a few OWASP members to provide coverage of OWASP. I am happy to enter into further discussions with OWASP members, but if you want me to build stuff then I am going to have to find a way of funding my time.
I'm certainly happy to acknowledge that this version makes the limitations of the Top Ten approach much clearer than previous versions, and explicitly encourages organizations to "think beyond the ten risks here". The document is careful not to claim the Top Ten as a full application security program, and warns readers not to stop at ten, because "there are hundreds of issues that could affect the overall security of a web application". But then surely this implies we shouldn't be wasting time reading this document at all; we should be reading the OWASP Developer’s Guide, "which is essential reading for anyone developing web applications today".
The status of the top ten items as risks (rather than, say, weaknesses or vulnerabilities or threats) is also a bit clearer, and the ranking of risks is based on the scale of the risk, not just the frequency of the attack. However, the document also refers to "relatively simple security problems like those in the OWASP Top 10" - which makes it seem that they may be the most obvious rather than the most problematic. Making people aware of simple problems doesn't necessarily promote awareness of more complex problems.
To my mind, the trouble with this kind of list is that it encourages bad thinking. Not only are some risks regarded as more attention-worthy than others (based on a generalized model of risk that may not be relevant to your organization or application portfolio), but each risk is considered in isolation. But a holistic understanding of security and risk needs to look at the composition of risk - how can several apparently small risks sometimes be multiplied into a very large risk.
I'm also concerned about limiting the analysis of risks to application security itself. Presumably a full security risk analysis would need to look at social attacks as well as technical attacks, but the Top Ten are all drawn from the technical side. I looked for this technical focus to be stated and explained somewhere, perhaps in a statement of scope, but couldn't find anything to this effect.
By the way, when I have raised issues about OWASP in the past, I have been challenged to fix them myself. But I'm not a normal member of OWASP, I'm an independent industry analyst who has been asked by a few OWASP members to provide coverage of OWASP. I am happy to enter into further discussions with OWASP members, but if you want me to build stuff then I am going to have to find a way of funding my time.
Should we take OWASP seriously?
Another stimulating discussion with @mcgoverntheory (James McGovern) about the ongoing OWASP project to identify the Top Ten Security Risks. I see no reason to change my previous opinion , which is that such lists are fundamentally misconceived.
As I've explained before (in this blog and elsewhere), I think the objectives of the list are muddled; I regard the methodology for producing the list as insufficiently rigorous; and I think it highly likely that the list will be widely used not as a precursor to a serious threat analysis but as a lazy substitute for it; so I just can't see that a Top Ten list is a good idea for anyone.
@mcgoverntheory replies "Many contributors to the top ten agreed that top ten lists as a concept are flawed. Its all about helping others move needle." Yes, but does it actually achieve any positive outcome? Show me.
@mcgoverntheory adds "Flawed concepts are propagated all the time. It's called marketing". But is it really the role of OWASP to be a marketing organization?
@mcgoverntheory continues "Everyone knows that Top X lists aren't meant to be complete nor necessarily measurable. Its about simple understanding". Well maybe everyone knows, but what matters is whether and how they act upon that knowledge.
@mcgoverntheory admits that "Sadly, most enterprises start and stop with awareness". Maybe so, but why should OWASP pander to this tendency?
And if OWASP is focusing its efforts on publicizing material that many contributors agree to be flawed, why on earth should industry analysts take OWASP seriously? Does OWASP want to be taken seriously?
Maybe it doesn't. @mcgoverntheory asks "What lift would analysts provide to OWASP? No products to sell and therefore we won't show up in quadrants or hype docs."
Of course, that depends what kind of industry analysis we are talking about. Some so-called industry analysis firms seem to do little more than reprocess and amplify the efforts of the software industry marketing departments, putting favoured products and vendors into a Magic Sorting Hat. Or they write like a theatre critic who gets invited to the previews, always finds something positive to say about the latest production, which can then be quoted on the play's website.
But I hope OWASP isn't the kind of organization that only wants analysis on its own terms, and understands that the value of industry analysis comes from the different perspective an analyst should be able to offer. In which case, I am happy to talk.
As I've explained before (in this blog and elsewhere), I think the objectives of the list are muddled; I regard the methodology for producing the list as insufficiently rigorous; and I think it highly likely that the list will be widely used not as a precursor to a serious threat analysis but as a lazy substitute for it; so I just can't see that a Top Ten list is a good idea for anyone.
@mcgoverntheory replies "Many contributors to the top ten agreed that top ten lists as a concept are flawed. Its all about helping others move needle." Yes, but does it actually achieve any positive outcome? Show me.
@mcgoverntheory adds "Flawed concepts are propagated all the time. It's called marketing". But is it really the role of OWASP to be a marketing organization?
@mcgoverntheory continues "Everyone knows that Top X lists aren't meant to be complete nor necessarily measurable. Its about simple understanding". Well maybe everyone knows, but what matters is whether and how they act upon that knowledge.
@mcgoverntheory admits that "Sadly, most enterprises start and stop with awareness". Maybe so, but why should OWASP pander to this tendency?
And if OWASP is focusing its efforts on publicizing material that many contributors agree to be flawed, why on earth should industry analysts take OWASP seriously? Does OWASP want to be taken seriously?
Maybe it doesn't. @mcgoverntheory asks "What lift would analysts provide to OWASP? No products to sell and therefore we won't show up in quadrants or hype docs."
Of course, that depends what kind of industry analysis we are talking about. Some so-called industry analysis firms seem to do little more than reprocess and amplify the efforts of the software industry marketing departments, putting favoured products and vendors into a Magic Sorting Hat. Or they write like a theatre critic who gets invited to the previews, always finds something positive to say about the latest production, which can then be quoted on the play's website.
But I hope OWASP isn't the kind of organization that only wants analysis on its own terms, and understands that the value of industry analysis comes from the different perspective an analyst should be able to offer. In which case, I am happy to talk.
Wednesday, December 09, 2009
IT suppliers face architectural risk
@tonyrcollins reports on the implications for large IT contracts of the Centrica v Accenture dispute (Computer Weekly, 9 December 2009). The dispute concerns a "best-of-breed" replacement billing system for the entire British Gas business, which Centrica ordered from Accenture in 2002.
Centrica is invoking a clause in the contract that refers to "fundamental defects", and a lot of the legal activity has been trying to determine what this phrase actually means. Although Accenture argues that the various problems experienced with the system have been unconnected and therefore don't count as fundamental, the High Court has accepted Centrica's interpretation that the cumulative effect of these defects may indeed be regarded as fundamental.
The article quotes Peter Clough, head of disputes at law firm Osborne Clarke:
So this is about architecture and risk. From a risk management perspective, a critical responsibility of the architect is to make sure that a lot of small problems don't add up to a big problem.
And it is also about procurement and risk. If this judgement stands, it appears to shift certain kinds of risk from the customer to the supplier. Obviously one solution to this would be to redraft procurement contracts. But another solution may be that large IT suppliers may be required to engage much more proactively with the broader architectural context for the systems they are building.
So can we expect all the major IT suppliers to look at architecture and risk from a new perspective?
Centrica is invoking a clause in the contract that refers to "fundamental defects", and a lot of the legal activity has been trying to determine what this phrase actually means. Although Accenture argues that the various problems experienced with the system have been unconnected and therefore don't count as fundamental, the High Court has accepted Centrica's interpretation that the cumulative effect of these defects may indeed be regarded as fundamental.
The article quotes Peter Clough, head of disputes at law firm Osborne Clarke:
"One of the important points to note about this case is that IT suppliers can be liable for claims for fundamental breach arising from the cumulative effect of a series of faults, each of which could look relatively minor in isolation. The majority of systems will of course be inter-linked so that a defect in part of the process could affect another part, snowballing into a more serious issue."
So this is about architecture and risk. From a risk management perspective, a critical responsibility of the architect is to make sure that a lot of small problems don't add up to a big problem.
And it is also about procurement and risk. If this judgement stands, it appears to shift certain kinds of risk from the customer to the supplier. Obviously one solution to this would be to redraft procurement contracts. But another solution may be that large IT suppliers may be required to engage much more proactively with the broader architectural context for the systems they are building.
So can we expect all the major IT suppliers to look at architecture and risk from a new perspective?
Friday, January 09, 2009
OWASP Top Ten - Update
OWASP is the Open Web Application Security Project. It is perhaps best-known for publishing Lists of the Top Ten (or more recently Top Twenty-Five) Security Bugs (or Vulnerabilities or Threats or Risks).
Following my earlier post on the OWASP Top Ten, as well as an exchange of emails with someone in the OWASP community, I posted the following question to the OWASP discussion group on Linked-In.
This prompted a couple of interesting responses, expressing different views on the real purpose of the OWASP Top Ten. Michael Vance said that the items in the top ten list are those most likely to occur or those that are most likely to have the greatest impact. Christian Frichot said that lists are good at removing the low hanging fruit: I interpret this as meaning the most obvious and easiest to fix, which is not necessarily the same as frequency or impact.
In any case, the methodology for creating the OWASP top ten list does not seem to be designed to produce a list with the characteristics required by either Michael or Christian. It is partly based on historical data (frequency but not impact or low-hangingness, as far as I can see), but with some adjustment to allow for some future projections of increased risk. For example, one issue (CSRF) was promoted to the list because the team believed it to be important, but with no evidence produced to support this belief. So is the OWASP Top Ten List really based on a systematic assessment of (generic) likelihood and impact?
In any case, it would be strange if the same list were equally relevant to all applications in all organizations. Do we expect a retail bank to have the same security risks as a nuclear power plant? Do we expect an airline to have the same security risks as an online bookstore?
Clearly it would be stupid to rely completely on the Top Ten List - although I suspect that some people do just that. But my question is more fundamental - what are the grounds for thinking that a top ten list improves the overall process, rather than just adding a redundant step into the process? Christian's argument is interesting - by dealing quickly with the easy and obvious generic vulnerabilities, we can spend more time on the specific ones. But is that what people actually do?
Michael acknowledges that there is a significant disconnect between the way that Top Ten (and Top 20 and Top 25 and even Threat Classification) lists should be used and the way that they are used. He mentions a specific concern that this list will be misused by being improperly inserted into procurement language.
If OWASP were merely an academic organization, it could deny responsibility for how other people use their lists. "We produce the perfect lists, it's not our fault if people abuse them." But if OWASP is trying to make a real practical difference to security, then the actual effects and effectiveness of these lists is important.
Meanwhile, I am happy to see that other security experts agree with my concerns. Gary McGraw (CTO of Cigital) has just published an excellent article called Software [In]security: Top 11 Reasons Why Top 10 (or Top 25) Lists Don’t Work (via Bruce Schneier).
Update (March 2009)
Tom Brennan has just posed a question on the Linked-In discussion: "So what OWASP project are you going to start that will change this?" So the way to influence existing projects within OWASP is to start a rival project is it? What a strange organization!
Related posts: OWASP Top Ten (October 2008), OWASP Top Ten 2010 (January 2010), Low-Hanging Fruit (August 2019)
Following my earlier post on the OWASP Top Ten, as well as an exchange of emails with someone in the OWASP community, I posted the following question to the OWASP discussion group on Linked-In.
Do Top-Ten Lists distract from a holistic approach to security?
If you ask people to pay attention to the top ten items in a list of threats or vulnerabilities, they will almost inevitably pay less attention to other things. (Intelligent people are aware of the limitations of lists, but even they are not immune to such effects.)
If a security vendor has a particular interest in one item - for example selling protection or detection for a particular threat - then there may be some commercial significance in whether that item makes the top ten or not. So a commercially minded security vendor will look for ways of influencing (aka distorting) the top ten list in his favour.
Meanwhile, intelligent attackers may calculate that a significant portion of security dollars will be consumed by the top ten, leaving other vulnerabilities under-funded.
The OWASP website does contain a page (Where To Go From Here) explaining that the top ten list is only the starting point of a proper security analysis, but this page is very poorly signposted and I suspect that many people never reach this page.
The official purpose of the OWASP list is to educate people about the consequences of security vulnerabilities. But I think there is a broader education purpose, and I fear that top ten lists distract from this purpose.
This prompted a couple of interesting responses, expressing different views on the real purpose of the OWASP Top Ten. Michael Vance said that the items in the top ten list are those most likely to occur or those that are most likely to have the greatest impact. Christian Frichot said that lists are good at removing the low hanging fruit: I interpret this as meaning the most obvious and easiest to fix, which is not necessarily the same as frequency or impact.
In any case, the methodology for creating the OWASP top ten list does not seem to be designed to produce a list with the characteristics required by either Michael or Christian. It is partly based on historical data (frequency but not impact or low-hangingness, as far as I can see), but with some adjustment to allow for some future projections of increased risk. For example, one issue (CSRF) was promoted to the list because the team believed it to be important, but with no evidence produced to support this belief. So is the OWASP Top Ten List really based on a systematic assessment of (generic) likelihood and impact?
In any case, it would be strange if the same list were equally relevant to all applications in all organizations. Do we expect a retail bank to have the same security risks as a nuclear power plant? Do we expect an airline to have the same security risks as an online bookstore?
Clearly it would be stupid to rely completely on the Top Ten List - although I suspect that some people do just that. But my question is more fundamental - what are the grounds for thinking that a top ten list improves the overall process, rather than just adding a redundant step into the process? Christian's argument is interesting - by dealing quickly with the easy and obvious generic vulnerabilities, we can spend more time on the specific ones. But is that what people actually do?
Michael acknowledges that there is a significant disconnect between the way that Top Ten (and Top 20 and Top 25 and even Threat Classification) lists should be used and the way that they are used. He mentions a specific concern that this list will be misused by being improperly inserted into procurement language.
If OWASP were merely an academic organization, it could deny responsibility for how other people use their lists. "We produce the perfect lists, it's not our fault if people abuse them." But if OWASP is trying to make a real practical difference to security, then the actual effects and effectiveness of these lists is important.
Meanwhile, I am happy to see that other security experts agree with my concerns. Gary McGraw (CTO of Cigital) has just published an excellent article called Software [In]security: Top 11 Reasons Why Top 10 (or Top 25) Lists Don’t Work (via Bruce Schneier).
Update (March 2009)
Tom Brennan has just posed a question on the Linked-In discussion: "So what OWASP project are you going to start that will change this?" So the way to influence existing projects within OWASP is to start a rival project is it? What a strange organization!
Related posts: OWASP Top Ten (October 2008), OWASP Top Ten 2010 (January 2010), Low-Hanging Fruit (August 2019)
Thursday, October 23, 2008
OWASP Top Ten
Back in August, James McGovern asked me to provide some OWASP coverage. Someone called Jennifer (Bayuk perhaps?) added a comment
Okay, let me start from that point. The OWASP Top Ten Project periodically publishes a "Top Ten" list of the most common web application security vulnerabilities. The official purpose of this list is to educate people about the consequences of these vulnerabilities.
But of course the inevitable effect of publishing a Top Ten list is pretty obvious - it causes people to pay particular attention to the items in the top ten, and considerably less attention to the items that don't quite make the top ten. If I was a niche security vendor, I'd be lobbying extremely hard to make sure that the particular vulnerability addressed by my product got into the top ten. Conversely, if I were running a criminal scam, I know exactly which vulnerabilities I'd be targeting.
This kind of thing clearly distracts people from a proper holistic view of application security. In my view it is the Top Ten List itself that is the "interesting distraction" Jennifer talks about, and I think OWASP should quietly drop this kind of cheap journalism and concentrate on educating people to do security properly. There is a lot of more intelligent stuff on the OWASP website explaining where to go from here, but I wonder how many people get that far?
Never let it be said that I am just a passive critic, however. Back in August, I registed onto the OWASP wiki and posted a couple of helpful questions about the OWASP principles. Haven't had a response yet, but I live in hope.
See also
OWASP Top Ten Update (January 2009)
OWASP Top Ten 2010 (January 2010)
OWASP is not dominated by commercial interests, and so the message is different than from product vendors (and service vendors too, to a lesser extent). When an automated tool vendor claims to "address" the OWASP Top Ten, they should be ashamed of themselves. And you should be ashamed if you're buying that hype and promoting automated tools as anything much more than an interesting distraction. Covering OWASP would allow people to get a far less biased opinion of what's going on in application security.
Okay, let me start from that point. The OWASP Top Ten Project periodically publishes a "Top Ten" list of the most common web application security vulnerabilities. The official purpose of this list is to educate people about the consequences of these vulnerabilities.
But of course the inevitable effect of publishing a Top Ten list is pretty obvious - it causes people to pay particular attention to the items in the top ten, and considerably less attention to the items that don't quite make the top ten. If I was a niche security vendor, I'd be lobbying extremely hard to make sure that the particular vulnerability addressed by my product got into the top ten. Conversely, if I were running a criminal scam, I know exactly which vulnerabilities I'd be targeting.
This kind of thing clearly distracts people from a proper holistic view of application security. In my view it is the Top Ten List itself that is the "interesting distraction" Jennifer talks about, and I think OWASP should quietly drop this kind of cheap journalism and concentrate on educating people to do security properly. There is a lot of more intelligent stuff on the OWASP website explaining where to go from here, but I wonder how many people get that far?
Never let it be said that I am just a passive critic, however. Back in August, I registed onto the OWASP wiki and posted a couple of helpful questions about the OWASP principles. Haven't had a response yet, but I live in hope.
See also
OWASP Top Ten Update (January 2009)
OWASP Top Ten 2010 (January 2010)
Tuesday, August 12, 2008
OWASP Coverage?
In a comment to an unrelated post, James McGovern asks
I can't speak for anyone else, but here's my answer. I might provide occasional comments about OWASP without any special motivation, but before I go to the trouble to provide comprehensive coverage about something, I need to see some strong interest from my readers. I also need to feel that this is a subject I can add some value to, rather than merely repeating what everyone else is saying.
So if anyone wants me to take a thorough look at OWASP (or anything else for that matter), please add a comment to this blog, indicating the nature of your interest and what specific questions you'd like me to address. Thanks.
"What would it take for an industry analyst to provide comprehensive coverage via blog entries on the work that OWASP is doing?"
I can't speak for anyone else, but here's my answer. I might provide occasional comments about OWASP without any special motivation, but before I go to the trouble to provide comprehensive coverage about something, I need to see some strong interest from my readers. I also need to feel that this is a subject I can add some value to, rather than merely repeating what everyone else is saying.
So if anyone wants me to take a thorough look at OWASP (or anything else for that matter), please add a comment to this blog, indicating the nature of your interest and what specific questions you'd like me to address. Thanks.
Sunday, May 18, 2008
Guardian Angel
From a recent US patent application
Twenty investors are listed, including Gates, William H. (Medina, WA) and Ozzie, Raymond E. (Seattle, WA). The presence of these two names on the patent application is attracting some attention from the blogosphere.An intelligent personalized agent monitors, regulates, and advises a user in decision-making processes for efficiency or safety concerns. The agent monitors an environment and present characteristics of a user and analyzes such information in view of stored preferences specific to one of multiple profiles of the user. Based on the analysis, the agent can suggest or automatically implement a solution to a given issue or problem. In addition, the agent can identify another potential issue that requires attention and suggests or implements action accordingly. Furthermore, the agent can communicate with other users or devices by providing and acquiring information to assist in future decisions. All aspects of environment observation, decision assistance, and external communication can be flexibly limited or allowed as desired by the user.
a most unusual Microsoft patent application that should intrigue privacy advocates
[TheoDP]interesting and frightening at the same time
[Dennis Kudin on security]This sounds interesting at first glance, but also a little creepy. ... I'm not so sure I'd be terribly keen on having my device capable of some of those functions.
[PDAPro.info]
There is some discussion in the comments to Bruce Schneier's blog about the extent of Bill's and Ray's contribution to this invention. Maybe it's true that Bill and Ray can attach their names to pretty much any Microsoft patent application if they choose. In which case the interesting question is what it was about this particular invention that attracted their interest.
The name Guardian Angel
is leading some commentators to view this as a security mechanism, but it is clearly intended to provide much more than security, a comprehensive mechanism to provide presence and context, which are key elements of some of the things both Bill and Ray have talked about in the past.
There is also some discussion on Bruce's blog about the originality of the invention and the possibility of prior art
. You really can't tell this from the summary though; to assess this properly, you would need to look at the whole application including the diagrams, but I haven't managed to access the diagrams. Clearly there are other companies working on mechanisms for presence and context, including the telecoms companies. I had a briefing on this very topic with Avaya recently. See my notes on Presence 2.0.
See also: What does a patent say? (February 2023)
Tuesday, August 28, 2007
Skype Skuppered
It turns out that it was Microsoft that brought down Skype for two days earlier this month. Microsoft's monthly software update (known as Patch Tuesday) triggered millions of computers to reboot at the same time, which always puts an unusual strain on major Internet companies such as Skype.
As Alex from RiskManagement Insight points out, this is equivalent to a form of DDOS (distributed denial of service) attack. From a risk management perspective, it may not matter very much whether an attack is deliberate and malicious, or merely an accidental side-effect of some entirely innocent action.
Although Skype had survived previous Patch Tuesdays without incident, it seems that this month's Patch Tuesday triggered a previously unknown bug in Skype's software. As Alex points out, it is practically impossible to construct a test environment large and complex enough to simulate this scenario.
I haven't seen any figures, but I have little doubt that Skype's competitors (including Microsoft) must have experienced an unusually high level of new registrations during Skype's misfortune. Now that we have become accustomed to free voice calls over the Internet, it seemed outrageous to return to the almost mediaeval practice of paying real money for talking over the telephone, so my colleagues and I signed up to Yahoo Messenger.
It's an ill wind ...
Source: The Register, August 20th 2007.
See also TechCrunch August 20th 2007.
See also TechCrunch August 20th 2007.
As Alex from RiskManagement Insight points out, this is equivalent to a form of DDOS (distributed denial of service) attack. From a risk management perspective, it may not matter very much whether an attack is deliberate and malicious, or merely an accidental side-effect of some entirely innocent action.
Although Skype had survived previous Patch Tuesdays without incident, it seems that this month's Patch Tuesday triggered a previously unknown bug in Skype's software. As Alex points out, it is practically impossible to construct a test environment large and complex enough to simulate this scenario.
I haven't seen any figures, but I have little doubt that Skype's competitors (including Microsoft) must have experienced an unusually high level of new registrations during Skype's misfortune. Now that we have become accustomed to free voice calls over the Internet, it seemed outrageous to return to the almost mediaeval practice of paying real money for talking over the telephone, so my colleagues and I signed up to Yahoo Messenger.
It's an ill wind ...
Friday, May 11, 2007
IT Security Industry
Lots of people (e.g. Gunner Petersen, Pete Lindstrom) are attacking Bruce Schneier for asking Do we really need a security industry?
Obviously Bruce doesn't expect the IT security industry to disappear any time soon. He points to some of the structural reasons for the economic viability of stand-alone products and services for IT security (including legal liability - or the lack of it), as well as the vested interests of software companies.
In some ways, the global security situation is getting worse with the increasing fragmentation of functionality and responsibility, and the increased interconnectedness of human and automated systems. This phenomenon isn't just an IT problem: it exists in other domains as well.
Bruce's argument is that security should be (increasingly) embedded into the infrastructure. This is the logic underlying the acquisition of Bruce's own company by BT last year. (See my comment: BT enters the Blogosphere.)
Pete is scornful, and and there are some similar comments on Bruce's own blog:
Do we need an IT security industry? Probably yes, but not the one we've got at the moment.
Obviously Bruce doesn't expect the IT security industry to disappear any time soon. He points to some of the structural reasons for the economic viability of stand-alone products and services for IT security (including legal liability - or the lack of it), as well as the vested interests of software companies.
In some ways, the global security situation is getting worse with the increasing fragmentation of functionality and responsibility, and the increased interconnectedness of human and automated systems. This phenomenon isn't just an IT problem: it exists in other domains as well.
Bruce's argument is that security should be (increasingly) embedded into the infrastructure. This is the logic underlying the acquisition of Bruce's own company by BT last year. (See my comment: BT enters the Blogosphere.)
Pete is scornful, and and there are some similar comments on Bruce's own blog:
"The notion of 'natural' security in the face of an intelligent adversary is so fundamentally ignorant that the whole thing must be a charade. It isn't even a pipe dream - it is an impossibility. Throw in the fact that IT resources are increasing in value and function and there is no doubt of that impossibility."Gunnar's criticisms are more moderate. He also questions the notion of natural security, but acknowledges the problems with the present situation:
"The way the IT security industry is presently constituted, is not effective, focuses WAY too much on network security instead of app and data security, and is incredibly reactive and tactically focused."For my part, I think it's always useful to ask provocative questions. Questions like "Do we really need X?" (or the equally provocative "Does Y matter?") shouldn't be dismissed with a simple Yes/No answer. Such questions call for an exploration of the true actual or potential value of X and Y, and perhaps a search for better (more innovative, more intelligent) alternatives to the current state-of-the-art.
Do we need an IT security industry? Probably yes, but not the one we've got at the moment.
Sunday, February 11, 2007
Problem-Solving
There are two contrasting patterns of problem-solving behaviour in the software industry.
I therefore find it odd that some classes of recurring problem continue to be tackled on a one-off basis. For example, the industry still doesn't seem to have found a reliable way to eliminate software code "overflows" - even though this is a regular cause of software bugs and security vulnerabilities.
Another common example of this pattern occurs in user support. When a user reports a problem, this probably indicates that a number of other users have a similar problem. And it is probably not good enough to fix the problem only for the users who report the problem. In fact it may be more important to fix the problem for those users who haven't noticed that there is a problem at all.
But if the response is to solve the problem as if it belonged to a single user, then this seems to deny the existence of a broader problem.
Take blog feeds for example. A couple of times recently I've noticed problems with blog feeds, and I've gone to the trouble to notify the blog author. What I'd expect the blog author to do is fix the feed. What happens instead is that the blog author sends me back a helpful email telling me how to redirect my newsreader. Actually I can work that out for myself thanks.
Perhaps some blog authors assume that their subscribers are all fluent in RSS. Because I'm the one identifying a problem, they might imagine I am positioning myself at the incompetent end of the spectrum. And the problem is my problem.
Actually, it's precisely because I'm not at the incompetent end of the spectrum that I can see there is a problem. And it's not my problem if the blog author loses some of his subscribers because his feed is broken. It's his problem.
- Solving problems on a one-off basis
- Solving an entire class of problems in a single move.
I therefore find it odd that some classes of recurring problem continue to be tackled on a one-off basis. For example, the industry still doesn't seem to have found a reliable way to eliminate software code "overflows" - even though this is a regular cause of software bugs and security vulnerabilities.
Another common example of this pattern occurs in user support. When a user reports a problem, this probably indicates that a number of other users have a similar problem. And it is probably not good enough to fix the problem only for the users who report the problem. In fact it may be more important to fix the problem for those users who haven't noticed that there is a problem at all.
But if the response is to solve the problem as if it belonged to a single user, then this seems to deny the existence of a broader problem.
Take blog feeds for example. A couple of times recently I've noticed problems with blog feeds, and I've gone to the trouble to notify the blog author. What I'd expect the blog author to do is fix the feed. What happens instead is that the blog author sends me back a helpful email telling me how to redirect my newsreader. Actually I can work that out for myself thanks.
Perhaps some blog authors assume that their subscribers are all fluent in RSS. Because I'm the one identifying a problem, they might imagine I am positioning myself at the incompetent end of the spectrum. And the problem is my problem.
Actually, it's precisely because I'm not at the incompetent end of the spectrum that I can see there is a problem. And it's not my problem if the blog author loses some of his subscribers because his feed is broken. It's his problem.
Tuesday, May 03, 2005
Jericho
Fortress Security
Back in 2002, Aidan Ward and I wrote some reports for the CBDI Forum on Web Services Security, which among other things lay siege to the Fortress Model of security. We were ahead of our time. The Fortress walls are not crumbling yet, but we are now joined by some serious allies.
See also brief note on Autonomous Computing: Fiefdoms & Fortresses
Jericho Forum
Jericho Forum (part of the Open Group) is a non-profit security standards group, led by user organizations. This is leading the push towards more agile and interoperable security models.
Press Release: Executives Agree that Interoperability, Deperimeterization of Data and Horizontal Integration Are Essential (April 2004) News Story: New boundaries and new rules (John Sterlicchi, SC Magazine, Jan 2005) News Story: Vendors line up to see Jericho vision (Ron Condon, SC Magazine, Feb 2005) News Story: The Future of IT Security is Fewer Walls, Not More (Dan Ilett, ZDNet, April 2005)
dePerimeterization
This essentially means tearing down the Fortress model. Definitions: Whatis.com, Word of the DaySecurity Vendors
| nCipher | Cryptographic IT Security | See press release (April 2005), on joining the Jericho Forum. |
| Vordel | XML Web Services Security | See weblog postings (March 2004, July 2004) by CTO Mark O'Neil |
CBDI Forum
Web Service Security (CBDI Journal, January 2002)
Component-Based Security for Web Services (CBDI Special Report, July 2002)
Agile Security for SOA (CBDI Journal, June 2005)
Subscribe to:
Posts (Atom)