Anne Jonas & Jenna Burrell's "Friction, snake oil, and weird countries: Cybersecurity systems could deepen global inequality through regional blocking" (2019) (link)
p.3
We draw insight from critiques of globalization and postcoloniality that decenter this normative ‘‘user’’ pointing to the way marginality is easily cast as aberrant and illegitimate (Burrell, 2012; Dourish and Mainwaring, 2012; Irani et al., 2010). We have worked with research collaborators exploring the question of regional blocking in parallel, but from within methodological traditions in computer science (see Afroz et al., 2018). In their work, they set about validating the existence and prevalence of ‘‘server-side’’ regional blocking utilizing automated techniques. Our approach pursued another set of questions. Defined by Irani et al.’s (2010) framework of postcolonial computing, ‘‘a project of understanding how all design research and practice is culturally located and power laden,’’ we sought to critically examine the cultural assumptions of blocking technologies and how they are influenced by the particular social locations of professionals who develop and deploy such practices. [emphases added by danielsgriffin]
p.3-4
The literature on fairness in machine learning has especially considered application domains such as criminal justice, lending, and social services where mechanisms of allocation impinge upon civil rights. Cybersecurity has been largely left out of this examination, though it is a key application area for machine learning that is widely applied in spam filtering and fraud detection. A key question is whether such systems work as well for users from one part of the world as they do in another. Are false positives (users identified as ‘bad’ actors who are in fact legitimate users) more prevalent in certain geographies or among certain subpopulations? Furthermore, could the definition of what constitutes legitimate use be reconsidered? These questions resonate with literature in predictive policing and algorithmic identification of child exploitation, where concerns about disparate impact and harm on marginalized communities are crowded out by public safety rationales (Selbst, 2017) and ‘‘proximity to innocence’’ (Thakor, 2018). We take up these questions by examining the framings of those who design, use, and request such systems, exploring how built in expectations about users have the potential to produce unintended consequences. [emphases added by danielsgriffin]
p.10
§ Removing those who cannot conform
Regional blocking can also be ‘‘effective’’ not only by preventing fraud, but by removing data that could complicate the smooth functioning of algorithmic recognition and sorting practices—taking away ‘‘friction’’ for those users who are considered valuable. At first, the refusal to collect data on people from blocked countries by refusing to provide services to them seems antithetical to the drive described by Fourcade and Healy to capture as much behavioral data as possible to sort people into appropriate categories. But by turning away those who might foreground the limitations of the algorithms, these systems can then boast higher rates of accuracy for those who are allowed in—those who alreadymatch pre-determined criteria. As Dourish and Mainwaring (2012) argue, ubiquitous computing has frequently obscured ‘‘particular geographical, institutional, commercial, and historical settings,’’ claiming to develop ‘‘global’’ solutions that actually just re-enforce the dominance of one ‘‘global authority’’ over all other ways of being. Thus websites that want to serve a particular group of people while claiming omnipotence are better positioned to do so if they eliminate the anomalies of people from places less prone to producing profits. [emphases added by danielsgriffin]