This is the first in a series of posts looking at the crisis of intelligibility in modern computer systems, and the threat that this poses to individual empowerment if we don’t get to grips with it.
As expected, the final text of Europe’s new Data Protection Regulation puts more emphasis data subject consent than the previous Directive. There’s a legitimate debate around whether consent is the right approach to data protection, or whether it’s just a distraction from more effective regulation, but in this series of posts I want to explore a broader, but related, and important, but often ignored, issue: The very real problem of technological intelligibility and the risk of technology disempowering everybody.
It should be stated, so I will, that my opinion is shaped hugely by a liberal philosophy; I like the idea of consent, and the EU approach to Data Protection precisely because it gives consumers rights over their personal data, rather than setting absolute limits on precisely what service providers can or can’t do. In Europe, if you consent to your data being processed then it’s pretty much fair game. If you want to find out how it’s being processed, or what is held, or challenge that processing, though, then you (as a human being, not necessarily an EU citizen) have a set of rights to help you do so. This appeals to me because it empowers individuals as intelligent (if not necessarily rational) agents. If you want to sell your genome because an advertising company offers you a vast sum of money for it, you can do so; but if you don’t want an advertising company to process your genome then you have the right to challenge it if they try.
To me, as a liberal (with a big and a little L), individual empowerment – the ability for individuals to choose and to shape their own lives – is of fundamental importance. To me that means challenging state power and social inequality, as well as other factors such as patriarchy and, as I’ll explore through these blog posts, the ways in which technology shapes our opportunity, choices and lives.
I’m a technologist by training, and I do believe that it has the potential to improve the lives of human beings and to create a fairer and more liberal society. I don’t think that is a given, though; technology can obviously be a distraction from problems, an ineffective smokescreen that gives the appearance of doing something without actually helping, or it can actively work against our interests. This critical viewpoint is one that the industry as a whole (and an often sycophantic media) often ignores, and is something I’ve tried to champion while an editor of XRDS magazine. My work as part of the meaningful consent project, and my PhD thesis, has brought me to the conclusion that we, as an industry and a society, may be stumbling blindly towards a future in which the potential of digital technology is missed, and which instead of supporting citizens to reach their potential fundamentally disempowers them.
For a long time, we’ve trumpeted the idea of advanced technology being indistinguishable from a magic as an achievement, as an implicit design goal or something to strive for; the technology industry is largely proud of being ‘magical’. Magic, though, is almost by definition the preserve of a few ‘gifted’ individuals, at best unintelligible to most and at worst completely unavailable to them. Magic is about power, that’s why it makes for exciting stories; and it is about being mysterious, which is why those stories needn’t explain how it actually works. Our most compelling stories about magic inextricably link magical ability to the antagonist – whether it’s as a tool to undermine the autonomy of others (as Sauron attempts in the Lord of the Rings), a justification to subject non-magical others to your whim (as Voldemort in Harry Potter) or a malevolent influence in and of itself (as the Dark Side of the Force in Star Wars). Basically, magic never goes well.
I think that the unintelligibility of magic is what makes it so troublesome. It’s the unintelligibility that makes is both unpredictable and exclusive. If we can’t predict, we can’t shape our environment to our own ends. If something is exclusive then those with access to it have a disproportionate power over everyone else. Magical technology that we don’t understand, cannot predict the consequences of, and do not have the ability to master does not empower us. At best it just is and at worst it shapes our lives as a result of someone else’s intent (or inattention).
Nowhere is this better demonstrated than in the web’s ubiquitous advertising network. Natasa Milic-Frayling (then at Microsoft Research), Eduarda Mendes Rodrigues (then at the University of Porto), m.c. schraefel (my academic supervisor) and I demonstrated the sheer extent of the tracking to which we are subject in our 2013 paper. Using search engines as an entry point into the Web we crawled thousands of pages to uncover the invisible network of advertisers, brokers and content providers that work together to collect information about the web pages we visit and to transfer that data among themselves to deliver advertisements individually tailored to our supposed interests. It takes just 30 clicks before the average web user has a 95% chance of being labelled by all of the top 10 tracking domains. Despite the invisibility of these networks themselves, some of the organisations participating in them are household names; Google, Facebook, Twitter – all are deeply implicated, and all have access to far more data about us than merely what we type into their own websites.
The ability for a website we’ve never visited before to deliver an advertisement tailored to our interests, or to a status update we posted “privately” to Facebook the previous day, even to our household income or our credit rating, is magical. Most of us probably wouldn’t say it was magical in a good way, more at the Voldemort end of the scale than the Gandalf side of things; it is magical nonetheless. In particular, it is unintelligible to most of us, even the websites that benefit from the advertising revenue. In fact, the complexity of the emergent network itself means that the actual extent of the data brokerage is probably beyond the understanding of most of the organisations involved in it.
The unintelligibility of the advertising network makes it virtually impossible to understand what the profiles it has created for each of us contain, or how we can influence them. This does not make for positive or fulfilling experiences; most web users perceive aggressive ad targeting as creepy or downright disturbing. Despite having dedicated years of research to the topic, I am still unable to account for some of the targeting that’s apparent when I browse the web, and I’m still unable to prevent much of it from taking place. We are all hugely disempowered by the existence of this “grey web”; we can’t opt out even if we want to. It is fortunate that, with some notable exceptions, despite the creepiness it is largely not a major threat to most of us.
Still, the grey web is just one way in which most of us have become (or at least feel that) we are pretty disempowered when it comes to exercising control over our personal data. Only yesterday the DVLA sent my driving license renewal letter to my flat in Southampton; not the address that appears on my license (which is registered to my parents’ house) after apparently checking “the last address you gave us with records held by a commercial partner.” How I would correct that record if it were wrong is anybody’s guess – There’s no insight into the magical process that they used and their description seems to provide little clue as to what they actually did or who they asked.
In this series of (hopefully weekly) posts I’ll expand on the idea of technology as a disempowering force, covering the need to make empowerment part of the standard design vernacular and how we might do so. Subscribe to the RSS feed, or follow @richardgomer on Twitter to make sure you don’t miss the next post!