Opening the Black Box: Analytics and Admissions

@peteyMIT via @EthanZ explains how technology is changing the university admissions process.

When kids apply to university in the USA, it is becoming increasingly common to include a link with supplementary information about the applicant - for example a project tumblr, a YouTube video, a Flickr album of artwork. The links are typically coded to track visitors, giving the applicant some idea about the level of interest the universities are showing. Chris Peterson finds this an uncomfortable experience: "As admissions officers, we are accustomed to reading applications; now, applications are reading us. ... Applicants are now armed with unprecedented insight into the processes that decide their fate."

There are several problems with this. Applicants and their parents may be misled by the tracking signals collected by these digital supplements, which may yield an entirely false picture of the university process. And yet applicants may attempt to use these signals as evidence that an application has not been properly considered. Even if the university attempts to block the analytics, this may still send the wrong message. (The absence of a signal is still a signal.)

In the past, analytics were a tool used by large organizations to monitor and control their customers. We are now seeing analytic platforms that seem to allow customers to monitor and control large organizations. Large organizations now need to understand how much information they are exposing to these platforms, and what conclusions their customers may draw. We can expect similar examples to appear in many other sectors.


Chris Peterson, Opening the Black Box: Analytics and Admissions (Chronicle of Higher Education, January 2013)

Updated 25 June 2015

Does Cameron's Dashboard App Improve the OrgIntelligence of Government?

In November 2012, it was announced that a mobile app to aid in decision-making and day-to-day government affairs was being trialled by the prime minister.

UK Trialling App to Aid Government Decisions (BBC News 8 November 2012)

Here are some quick comments from Twitter

@lesteph PM's dashboard is at best pointless, at worst dangerous, unless his briefing system has fundamentally collapsed

@dominiccampbell he may as well have it, but pretending it's anything other than a partial view and mostly for PR is daft

@willperrin rather an antediluvian counsel of despair there then. back to 'ringbinders full of..' briefing

@6loss The "dashboard vs intelligence" debate? IMHO dashboards are useless without fast feedback on action.

In a subsequent discussion on Linked-In, @6loss and I discussed some of the intriguing questions raised by this news story.

Firstly, we were missing the imperative for real-time action and feedback. Obviously the Prime Minister needs to know whether job vacancies are going up or down, but the idea of real-time update is just ridiculous. Suppose that seventeen new job vacancies have been posted in Smartchester in the past twenty minutes, Are we supposed to believe that these seventeen vacancies urgently need to be communicated to the PM so that he can take appropriate action?

What does make sense is a dashboard that supports an OODA loop. A well-designed dashboard should not only provide aggregated data, but also provide some way of making sense of the data. (It is possible that the data visualization may help here.) And then taking rapid action.

But in a well-designed organization, the responsibility for rapid action is delegated to the people in the front line, who are given the real-time intelligence and the resources/tools and the authority to solve problems effectively and efficiently. This is what the military call "Power to the Edge". A completely different order of intelligence is required at the centre, usually operating at a much slower tempo.

And since managers are often tempted to meddle with randomly varying processes (Deming called this "tampering"), a well-designed control system deliberately hides much of the volatility from senior management. (In cybernetics, this is called "attenuation".)

Secondly, I'm wondering what kind of statistics we are talking about here. When people talk about "statistics", they often mean the kind of statistics kids learn in primary school (totals and averages) rather than the kind of statistics kids learn in high school (correlation and significance). I wonder how many ministers could cope with high school statistics (let alone degree level) without a civil servant or adviser there to explain it to them? The danger of the "dashboard" is that it may eliminate the vital step of interpretation and sense-making, which is surely essential to evidence-based management. 

Thirdly, I'm wondering about the planned rollout of this App. Are we to suppose that all ministers and senior civil servants are going to be watching the same set of indicators, or does collective responsibility entail that each minister is watching a different set of indicators? In a typical control room, there are many people each watching a different dashboard or controlling a different sector: it would seem a bit redundant if they were all watching the same one. Meanwhile, the supervisor sits in his cubicle playing Angry Birds, or sending texts to his neighbours.

A few weeks after this discussion, writing in the New York Times, Will Wiles compared this dashboard with the Viable Systems Model implementation in Chile under Salvador Allende. He pointed out that the dashboard is not truly cybernetic because it lacks a mechanism to translate all that data into action. Quite so.

Will Wiles, Before Fruit Ninja, Cybernetics (New York Times, 30 November 2012)

 

Update: See also Shannon Mattern, Mission Control: A History of the Urban Dashboard (Places Journal, March 2015)