8
In mid-2019, an investigative journalism/tech non-profit called MuckRock, and Open the Government (OTG), a non-partisan advocacy group, began submitting freedom of information requests to law enforcement agencies across the United States. The goal: to smoke out details about the use of an app rumoured to offer unprecedented facial recognition capabilities to anyone with a smart phone.
Co-founded by Michael Morisy, a former Boston Globe editor, MuckRock specializes in freedom of information (FOIS) requests, and its site has grown into a publicly accessible repository of government documents obtained under access-to-information laws.
As responses trickled in, it became clear that the MuckRock/OTG team had made a discovery about a tech company called Clearview AI. Based on documents obtained from Atlanta, OTG researcher Freddy Martinez began filing more requests and discovered that as many as two hundred police departments across the U.S. were using Clear-view’s app, which compares images taken by smart phone cameras to a sprawling database of 3 billion open-source photographs of faces linked to various forms of personal information (e.g., Facebook profiles). It was, in effect, a point-click-and-identify system that radically transformed the work of police officers.
The documents soon found their way to a New York Times reporter named Kashmir Hill, who, in January 2020, published a deeply investigated feature about Clearview, a tiny and secretive start-up with backing from Peter Thiel, the Silicon Valley billionaire behind Paypal and Palantir Technologies. Among the story’s revelations, Hill disclosed that tech giants like Google and Apple were well aware that such an app could be developed using artificial intelligence algorithms feeding off the vast storehouse of facial images uploaded to social media platforms and other publicly accessible databases. But they had opted against designing such a disruptive and easily disseminated surveillance tool.
The Times story set off what could best be described as an international chain reaction, with widespread media coverage about the use of Clearview’s app, followed by a wave of announcements from various governments and police agencies about how Clearview’s app would be banned. The reaction played out against a backdrop of news reports about China’s nearly ubiquitous facial recognition–based surveillance networks.
Canada was not exempt. To Surveil and Predict, a detailed examination of ‘algorithmic policing’ published in fall 2020 by the University of Toronto’s Citizen Lab, noted that officers with law enforcement agencies in Calgary, Edmonton, and across Greater Toronto had tested Clearview’s app, sometimes without the knowledge of their superiors. Investigative reporting by the Toronto Star and Buzzfeed News found numerous examples of municipal law enforcement agencies, including the Toronto Police Service, using the app in crime investigations. The RCMP denied using Clearview even after it had entered into a contract with the company – a detail exposed by Vancouver’s The Tyee.
With federal and provincial privacy commissioners ordering investigations, Clearview and the RCMP subsequently severed ties, although Citizen Lab noted that many other tech companies still sell facial recognition systems in Canada. ‘I think it is very questionable whether [Clearview] would conform with Canadian law,’ Michael McEvoy, British Columbia’s privacy commissioner, told the Star in February 2020.
There was more fallout elsewhere. Four U.S. cities banned police use of facial recognition outright, the Citizen Lab report noted. The European Union in February proposed a ban on facial recognition in public spaces but later hedged. A U.K. court in April ruled that police facial recognition systems were ‘unlawful,’ marking a significant reversal in surveillance-minded Britain. And the European Data Protection Board, an EU agency, informed commission members in June 2020 that Clearview’s technology violates pan-European law enforcement policies. As Rutgers University’s Ellen Goodman comments, the use of data-intensive policing technologies has generated ‘huge blowback.’
There’s nothing new about surveillance or police investigative practices that draw on highly diverse forms of electronic information, from wiretaps to bank records and images captured by private security cameras. As early as the 1960s, a U.S. consulting firm, Simulmatics Corp., built computer simulations that purported to have the capacity to predict race riots based mash-ups of crime, demographic, and socio-economic data sets (Lepore 2020, 262). Yet during the 2010s, dramatic advances in big data analytics, biometrics, and AI, stoked by venture capital and law enforcement agencies eager to invest in new technology, have spurred on a fast-growing, well-capitalized industry that amasses huge collections of data, much of it open-source, to create powerful new tools for law enforcement agencies. As the Clearview story showed, regulation and democratic oversight have lagged far behind the technology.
U.S. start-ups like PredPol and HunchLab, now owned by ShotSpotter, a publicly traded company, have designed so-called ‘predictive policing’ algorithms that use law enforcement records and other geographical data (e.g., locations of schools) to make statistical guesses about the times and locations of future property or violent crimes. Palantir’s law-enforcement service aggregates and then mines huge data sets consisting of emails, court documents, evidence repositories, gang member databases, automated licence plate readers, social media, etc., to find hidden correlations or patterns that police can use to investigate suspects.
Sarah Brayne, a B.C.-born University of Texas at Austin sociologist and author of a 2021 book entitled Predict and Surveil: Data, Discretion, and the Future of Policing, observes that big data’s ‘appeal stems from its aura of objectivity.’ Echoing the safety promises advanced by autonomous vehicle advocates, Brayne observes that proponents of big data policing say the use of automated systems and huge sets of data removes the kind of bias and human error that leads to problematic policing practices, such as over-enforcement in low-income neighbourhoods. It is, in other words, the exemplar of evidence-based decision-making.
Yet as the Clearview fallout indicates, these systems are rife with technical, ethical, and political landmines. As Andrew G. Ferguson, a University of the District of Columbia law professor and authority on the subject, explained in his 2017 book, The Rise of Big Data Policing: Surveillance, Race, and the Future of Law Enforcement, analysts have identified an impressive list: biased, incomplete, or inaccurate data, opaque technology, erroneous predictions, lack of governance, public suspicions about surveillance and overpolicing, conflicts over access to proprietary algorithms, unauthorized use of data, and the muddied incentives of private firms selling law enforcement software. Further, as Brayne added in her study, human discretion and bias is baked into these technologies via programming, data entry, and the weighting assigned to different kinds of results (Brayne 2020, 100). The fingerprints of their creators have not been scrubbed away.
Brayne’s research revealed that police officers were highly skeptical of policing algorithms.13 Other critics point out that by deploying smart city sensors or other data-enabled systems, like transit smart cards, local governments may be inadvertently providing the police with new intelligence nodes. Metrolinx, the transit agency for the Greater Toronto and Hamilton Area, has released Presto card user information to police on a number of occasions, while London’s Metropolitan Police has made thousands of requests for Oyster card data to track criminals. ‘Any time you have a microphone, camera or a live-feed, these [become] surveillance devices with the simple addition of a court order,’ adds New York civil rights lawyer Albert Cahn, executive director of the Surveillance Technology Oversight Project.
The authors of the Citizen Lab study, lawyers Kate Robertson, Cynthia Khoo, and Yolanda Song, argue that Canadian governments need to impose a moratorium on the deployment of algorithmic policing technology until the public policy and legal frameworks can catch up.
Data policing was born in New York City in the early 1990s when then-police commissioner William Bratton launched CompStat, a computer system that compiled up-to-date crime information – incidents, times, locations, etc. – then visualized the findings in ‘heat maps.’ These allowed unit commanders to deploy officers to neighbourhoods most likely to be experiencing crime problems.
Originally conceived as a management tool that would push a demoralized police force to make better use of limited resources, CompStat is credited by some as contributing to the marked reduction in crime rates in the Big Apple, although many other big cities experienced similar drops through the 1990s and early 2000s.
The 9/11 terrorist attacks sparked enormous investments in security technology. The past two decades have seen the emergence of a multibillion-dollar industry dedicated to civilian security technology – everything from large-scale deployments of CCTVs and cyber-security to the development of highly sensitive biometric devices – fingerprint readers, iris scanners, etc. – designed to bulk up the security around factories, infrastructure, and government buildings.
Predictive policing and facial recognition technologies evolved on parallel tracks, both relying on increasingly sophisticated analytics techniques, artificial intelligence algorithms, and ever deeper pools of digital data.
The core idea is that the algorithms – essentially formulas, such as decision trees, that generate predictions – are ‘trained’ on large tranches of data so they become increasingly accurate – for example at anticipating the likely locations of future property crimes or matching a face captured in a digital image from a CCTV to one in a large database of headshots. Some algorithms are designed to use a set of rules with variables (akin to following a recipe). Others, known as machine learning systems, are programmed to learn on their own (trial and error).
The risk lies in the quality of the data used to train the algorithms – what was dubbed the ‘garbage-in-garbage-out’ problem in a study by the Center on Privacy and Technology at Georgetown Law. If there are hidden biases in the training data – e.g., it contains mostly Caucasian faces – the algorithm may misread Asian or Black faces and generate ‘false positives,’ a well-documented shortcoming if the application involves identifying a suspect in a crime.
Similarly, if a poor or racialized area is subject to overpolicing, there will likely be more crime reports, meaning the data from that neighbourhood is likely to reveal higher-than-average rates of certain types of criminal activity, a data point that would justify more over-policing and racial profiling. Some crimes, in turn, are underreported, and so don’t influence these algorithms or amplify feedback loops. Brayne, however, points out that big data analytics can ‘retrospectively’ diagnose problematic policing tactics, such as carding, through statistical analysis that can show proof of overpolicing in racialized neighbourhoods (Brayne 2020, 104).
Other predictive- and AI-based law enforcement technologies, including ‘social network analysis’ (SNA) – the notion that an individual’s web of personal relationships, gleaned, for example, from open sources such as social media platforms or the cross-referencing of lists of gang members – promised to generate predictions that individuals already known to police were at risk of becoming embroiled in shootings and other violent crimes. Yet, as Brayne points out, when police rely on big data analytics to generate leads, there’s a built-in incentive to feed the maw. As she related, during the years when she was embedded with the LAPD, she constantly found herself watching police officers aiming to ensure that seemingly innocuous details about individuals they stopped on the street would get ‘into the system,’ in the expectation that all the little shards of information may eventually form a pattern that powerful analytics technologies, such as those used by Palantir, will detect.
This type of sleuthing certainly seemed to hold out some promise. In one study, criminologists at Cardiff University found that ‘disorder-related’ posts on Twitter reflected actual crime incidents in metropolitan London – a finding that suggests how big data can help map and anticipate criminal activity. In practice, however, such surveillance tactics can prove to be explosive, as happened in 2016, when U.S. civil liberties groups revealed FOI documents showing that Geofeedia, a location-based data company, had signed contracts with numerous police departments to provide analytics based on social media posts to Twitter, Facebook, Instagram, etc. Among the individuals targeted by Geofeedia’s data: protesters and activists. Chastened, the social media firms rapidly blocked Geofeedia’s access.
The Chicago Police Department in 2013 began experimenting with predictive models that assigned risk scores for individuals based on criminal records or their connections to people involved in violent crime. By 2019, the CPD had assigned risk scores to almost 400,000 people and claimed to be using the information to surveil and target ‘at-risk’ individuals (including potential victims) or connect them to social services, according to a January 2020 report by Chicago’s inspector general.
These tools can draw mistaken or biased inferences in the same way that over-reliance on police checks in racialized neighbourhoods merely results in what could be described as guilt by address. The Citizen Lab study noted that the Ontario Human Rights Commission identified social network analyses as a potential cause of racial profiling. In the case of the CPD’S predictive risk model, the system was discontinued in 2020 after media reports and internal investigations showed that the police were adding people to their list based solely on arrest records, meaning they may not even have been charged, much less convicted of a crime (Foody 2020).
Early applications of facial recognition software included passport security systems or searches of mug-shot databases. But in 2011, the Insurance Corporation of B.C. offered Vancouver police the use of facial recognition software to match photos of Stanley Cup rioters with driver’s licence images – a move that prompted a stern warning from the province’s privacy commissioner. In 2019, the Washington Post revealed that the FBI and ICE investigators regarded state databases of digitized driver’s licences, which had been scanned without consent, as a ‘gold mine for facial recognition photos.’
In 2013, Canada’s federal privacy commissioner released a report on police use of facial recognition that anticipated the issues raised by Clearview app earlier this year: ‘[S]trict controls and increased transparency are needed to ensure that the use of facial recognition conforms with our privacy laws and our common sense of what is socially acceptable.’ (Canada’s data privacy laws are only now being updated.)
The technology, meanwhile, continues to gallop ahead. New York civil rights lawyer Albert Cahn points to the emergence of ‘gait recognition’ systems, which use similar visual analysis techniques to identify individuals by their walk; these systems are reportedly in use in China. ‘You’re trying to teach machines how to identify people who walk with the same gait,’ he says. ‘Of course, a lot of this is completely untested.’
The predictive policing story has evolved somewhat differently. The methodology grew out of analysis commissioned by the Los Angeles Police Department in the early 2010s. Two data scientists, P. Jeffrey Brantingham and George Mohler, used mathematical modelling to forecast copycat crimes based on data about the location and frequency of previous burglaries in three L.A. neighbourhoods. They published their results and soon set up PredPol to commercialize the technology. Media attention soon followed, as news stories played up the seemingly miraculous power of a Minority Report –like system that could do a decent job anticipating incidents of property crime.
Operationally, police forces used PredPol’s system by dividing up precincts in 150 square-metre ‘cells’ that police officers were instructed to patrol more intensively during periods when PredPol’s algorithm forecast criminal activity. In the post-2009 credit crisis period, the technology seemed to promise that cash-strapped American municipalities would get more bang for their policing buck.
Other firms, from start-ups to multinationals like IBM, entered the market with innovations, incorporating other types of data, such as socio-economic data or geographical features, from parks and picnic tables to schools and bars, that may be correlated to elevated incidents of certain types of crime. The reported crime data is routinely updated so the algorithm remains current. (A short-lived smart phone app, initially called Ghetto Tracker and later re-dubbed Good Part of Town, used a Microsoft routing algorithm, plus user feedback on their personal sense of safety in a given location, to create ‘safe’ pedestrian ‘travel tools.’ It no longer exists [Tiku 2013].)
Police departments across the U.S. and Europe have invested in various predictive policing tools, as have several in Canada, including in Vancouver, Edmonton, and Saskatoon. Whether they have made a difference is an open question. As with several other studies, a 2017 review by analysts with the Institute for International Research on Criminal Policy, at Ghent University in Belgium, found inconclusive results: some places showed improved results compared to more conventional policing, while in other cities, the use of predictive algorithms led to reduced policing costs but little measurable difference in outcomes.
Revealingly, the city where predictive policing really took hold, Los Angeles, has rolled back police use on these techniques. In the spring of 2020, the LAPD tore up its contract with PredPol in the wake of mounting community and legal pressure from the Stop LAPD Spying Coalition, which had found that individuals who posed no real threat, mostly Black or Latino, were ending up on police watch lists because of flaws in the way the system assigned risk scores.
‘Algorithms have no place in policing,’ coalition founder Hamid Khan said in an interview in 2020 with MIT Technology Review. ‘I think it’s crucial that we understand that there are lives at stake. This language of location-based policing is by itself a proxy for racism. They’re not there to police potholes and trees. They are there to police people in the location. So location gets criminalized, people get criminalized, and it’s only a few seconds away before the gun comes out and somebody gets shot and killed.’ (Similar advocacy campaigns, including proposed legislation governing surveillance technology and gang databases, have been proposed for New York City.)
There has been one other interesting consequence: police resistance. Sarah Brayne spent two and a half years deeply embedded with the LAPD, exploring the reaction of law enforcement officials to algorithmic policing techniques by conducting ride-alongs as well as interviews with dozens of veteran cops and data analysts. In results published in 2021, Brayne and collaborator Angèle Christin observed ‘strong processes of resistance fueled by fear of professional devaluation and threats of performance tracking.’
Before shifts, officers were told which squares to drive through, when and how frequently, and the location of their vehicles was tracked by an onboard GPS device to ensure compliance. But Brayne found that some would game the system by turning off the tracking device, which they regarded with suspicion. Others just didn’t buy what the technology was selling. ‘Patrol officers frequently asserted that they did not need an algorithm to tell them where crime occurs,’ she noted.
Brayne says that police departments increasingly see predictive technology as part of the tool kit, despite questions about effectiveness or other concerns, like racial profiling. ‘Once a particular technology is created,’ she observed,’ there’s a tendency to use it.’ Brayne, however, added one other prediction, which has to do with the future of algorithmic policing in the post–George Floyd era – ‘an intersection,’ as she says, ‘between squeezed budgets and this movement around defunding the police.’
In July 2018, Toronto city council voted to ask the federal and provincial governments for $44 million for a long list of outlays meant to confront sharp increases in gun violence across the city. The funding was to go toward youth recreation and employment initiatives, various intervention efforts involving the police, and funding for children’s mental health services.
But council, after a divisive debate, also supported a request from the police for an extra $4 million for a lot more CCTVs, as well as the deployment of the gunshot detection technology ShotSpotter, which uses specialized microphones and sophisticated software to identify and locate the source of outdoor gunfire, then automatically alerts 911 (Toronto Police Services Board 2018). Among the most outspoken proponents: Mayor John Tory, who said he ‘strongly supported’ the ask.
Less than half a year later, however, Toronto Police abruptly backed out of the ShotSpotter deal, citing potential legal concerns. ‘They are not proceeding for the same reason many of us voted against it in the first place … an invasion of privacy, that there were severe risks around data collection and use,’ former city councillor Joe Cressy told the Globe and Mail at the time. ‘Frankly, it was a shiny object in a Robo-Cop-style of enforcement model that was intended in the midst of the summer of the gun to make us all feel better’ (Gray 2019).
Founded in the mid-1990s by three engineers, ShotSpotter has developed a technology that claims to distinguish between gunshots and other explosive sounds, like firecrackers or backfiring mufflers. When its hidden microphones pick up a sound, it is electronically analyzed, not just for its characteristics – pitch, reverberation, etc. – but also potential location.
The latter estimates are generated by triangulating sounds picked up by several of ShotSpotter’s acoustic sensors (each one is programmed to estimate the distance between the sensor and the gunshot based on the time it takes for the sound to reach the microphone). The company says the audio fingerprint from detected gunshots is immediately relayed to a high-tech data processing centre, where analysts check them out and notify police if they fit the acoustical profile of a weapon being discharged. The notifications to police come with data such as the calculated location of the sound and a time stamp.
The California-based company – which went public on NASDAQ in 2017 and had a market capitalization of US$340 million as of late 2021 – promotes its technology to municipalities, universities, and other campus-based institutions by asserting that most gunshots are either not reported or cannot be accurately located. The company’s website is packed with statistics about the impact of its technology in different cities, as well as other claimed benefits, such as reduced homicides, improved police response times, and decreased mortality of gunshot victims. The firm generates revenue through annual fees based on the size of the area to be covered.
In the mid-2010s, the company’s claims began to receive more focused attention from reporters and criminologists. A 2021 peer-reviewed study published in the Journal of Urban Health and led by a researcher from the Center for Gun Violence Preventation and Policy at Johns Hopkins Bloomberg School of Public Health looked at crime data from sixty-eight big cities gathered between 1999 and 2016, where ShotSpotter was used.
The study concluded that the presence of ShotSpotter had ‘no significant impact on firearm-related homicides or arrest outcomes’ and further pointed out that a much more determinative cause of gun-related deaths had to do with state-level firearms policies, such as right-to-carry laws or more demanding permit-based systems. The authors also pointed out that the stats cited by ShotSpotter in support of its claims had not been peer-reviewed or exposed to more ‘robust’ research evaluations (Doucette et al. 2021).
Other skeptical reviews also rolled in, including a 2021 evaluation by the City of Chicago’s inspector general of the Chicago Police Department’s use of ShotSpotter, at a cost of US$33 million per year. ‘Of the 41,830 ShotSpotter alerts that logged a police response, only 4,556 – 9.1% – resulted in viable evidence that a gun-related criminal offense had occurred,’ a summary of the IG’s report noted. The problem, the analysis found, was that police weren’t necessarily acting on those ShotSpotter leads, or at least bringing in potential perpetrators. As an editorial in the Chicago Tribune stated, ‘It’s abundantly clear that the city needs to take a hard look at the merit of ShotSpotter’ (Editorial Board 2021).
At around the same time, investigative reporting by Vice Motherboard and the Associated Press offered stories of police in various cities requesting ShotSpotter analysts to modify their reports, in some instances in relation to incidents when police officers discharged their guns, with those shots detected by the company’s monitors. Court documents obtained by Vice/AP ‘suggest that the company’s analysts frequently modify alerts at the request of police departments – some of which appear to be grasping for evidence that supports their narrative of events’ (Feathers 2021).
That wasn’t the only uncomfortable finding. The reporting pointed out that ShotSpotter hasn’t allowed independent verification of its accuracy claims, which have risen steadily over time. It also cited a case of a man wrongfully accused and then found guilty of murder based on ShotSpotter evidence; the conviction was eventually overturned when a court found that the prosecutors hadn’t produced enough evidence.
In its own case against ShotSpotter, the American Civil Liberties Union also claimed the company’s acoustic sensors tended to be disproportionately located in racialized low-income communities – a practice that ‘can distort gunfire statistics and create a circular statistical justification for over-policing in communities of [colour].’
ShotSpotter, however, pushed back against what it described as ‘false claims’ with libel suits and extensive rebuttals posted prominently on its website. CEO Ralph Clark strongly denied that the company’s technology relies on artificial intelligence algorithms to evaluate whether a sound picked up by its sensors comes from a gun or another source. Those judgments, he insisted, are made by human beings.
‘The technology is a focused tool – highly accurate and unbiased in delivering evidence of a gunshot incident, including recorded sound of gunfire and the time and location of a shooting,’ he wrote in an op-ed. ‘This highly objective and factual evidence is commonly used by courts and can be examined and tested by both the prosecution and defense. ShotSpotter evidence on its own has never been nor could never be responsible for the charging, arrest or conviction of anyone accused of a crime.’
While the company continues to generate millions in revenues and to add customers, its stock and its earnings slid throughout 2021 – an indication that skepticism about its brand of high-tech policing had penetrated financial markets.
One of the unanticipated by-products of Sidewalk Labs’ attempt to build a wired community on Toronto’s waterfront is that it seemed to prompt heightened scrutiny of the use of digital technology by the city’s police service, including facial recognition systems and predictive policing techniques. A Toronto Star poll conducted not long after the city abandoned its ShotSpotter deal found four in ten respondents mistrusted police use of facial recognition, but almost as many thought the technology would help fight crime. In early 2020, Toronto’s police chief halted the use of Clearview’s technology, and the civil oversight body moved to seek public input on policy and rules to guide the use of any future AI-based policing technology.
At the core of the debate is a basic public policy principle: transparency. Do individuals have the tools to understand and debate the workings of a suite of technologies that can have tremendous influence over their lives and freedoms, and potentially violate their civil rights?
It’s what Andrew Ferguson and others refer to as the ‘black box’ problem. The algorithms, designed by software engineers, rely on certain assumptions, methodologies, and variables, none of which are visible, much less legible, to anyone without advanced technical know-how. Many, moreover, are proprietary because they are sold to local governments by private companies. The upshot is that AI algorithms have not been regulated by governments or oversight bodies, despite their use by public agencies.
New York City Council tried to tackle this question in May 2018 by establishing an ‘automated decision systems’ (ADS) task force to examine how municipal agencies and departments use AI and machine learning algorithms. The task force was to devise procedures for identifying hidden biases and to disclose how the algorithms generate choices so the public can assess their impact. The group included officials from the administration of Mayor Bill de Blasio, tech experts, and civil liberties advocates. It held public meetings throughout 2019 and released a report in 2019. NYC was, by most accounts, the first city to have tackled this question, and the initiative was, initially, well-received.
Going in, Albert Cahn saw the task force as ‘a unique opportunity to examine how AI was operating in city government.’ But he describes the outcome as ‘disheartening.’ ‘There was an unwillingness to challenge the NYPD on its use of ADS.’ Some other participants agreed, describing the effort as ‘a waste.’
If institutional obstacles thwarted an effort in a government the size of the City of New York, what does better and more effective oversight look like? A couple of answers have emerged.
In his book on big data policing, Andrew Ferguson writes that local governments should start at first principles, and urges police forces and civilian oversight bodies to address five fundamental questions, ideally in a public forum:
· Can you identify the risks that your big data technology is trying to address?
· Can you defend the inputs into the system (accuracy of data, soundness of methodology)?
· Can you defend the outputs of the system (how they will impact policing practice and community relationships)?
· Can you test the technology (offering accountability and some measure of transparency)?
· Is police use of the technology respectful of the autonomy of the people it will impact?
These ‘foundational’ questions, he writes, ‘must be satisfactorily answered before green-lighting any purchase or adopting a big data policing strategy’ (Ferguson, 88).
In addition to calling for a moratorium and a judicial inquiry into the uses of predictive policing and facial recognition systems, the authors of the Citizen Lab report made several other recommendations, including the need for full transparency; provincial policies governing the procurement of such systems; limits on the use of ADS in public spaces; and the establishment of oversight bodies that include members of historically marginalized or victimized groups.
Other watchdog groups raised related concerns. The Women’s Legal Education and Action Fund has pointed out that AI algorithms are, by their nature, inequitable because they are trained on data sets that invariably contain distortions. As a result, these flaws can’t be managed away simply by better policy or technological tweaks. Moreover, when AI algorithms are developed by private companies subject only to market forces, the predictions they generate can be more vulnerable to legal challenges. ‘Having more accurate AI systems does not mitigate inequality,’ the group wrote in a 2021 legal brief submitted to the Toronto Police Services Board (Thomasen et al. 2021).
Sarah Brayne adds that the proliferation of digital sensors in public space (e.g., ShotSpotter mics, CCTVS, or automated licence-plate readers) and the growth of very large sets of accessible data have boosted the capability of police to steadily widen their surveillance activities. One question, she points out, is whether the mere presence of someone’s information in databases accessed by law enforcement agencies raises the prospect of unlawful searches, in the form of repeated police queries of large data sets.
Despite such warnings, the Canadian government has sought to develop a more intentional and transparent approach, which University of Ottawa law professor and privacy expert Teresa Scassa describes as highly promising.
The Treasury Board Secretariat’s ‘Directive on Automated Decision-Making,’ which came into effect in April 2019, requires federal departments and agencies, except those involved in national security, to conduct ‘algorithmic impact assessments’ (AIA) to evaluate unintended bias before procuring or approving the use of technologies that rely on AI or machine learning. The policy directs the federal government to publish AIAS, release software codes developed internally, and continually monitor the performance of these systems. In the case of proprietary algorithms developed by private suppliers, federal officials have extensive rights to access and test the software.
Scassa points out that the policy includes due-process rules and looks for evidence of whether systemic bias has become embedded in these technologies, which can happen if the algorithms are trained on skewed data. She also observes that not all algorithm-driven systems generate life-altering decisions, e.g., chatbots that are now commonly used in online application processes. But where they are deployed in ‘high impact’ contexts such as policing – e.g., with algorithms that aim to identify individuals caught on surveillance videos – the policy requires ‘a human in the loop.’
The directive, says Scassa, is getting interest elsewhere. Ellen Goodman, at Rutgers, is hopeful this approach will also gain traction in the U.S., where municipal police departments were far more aggressive in their acquisition of these kinds of technology tools. The upshot is that Ottawa’s low-key but thorough approach could point to a way for citizens to shine some light into the black box that is big data policing.
13. One contributing factor to that skepticism, Brayne explains, is that LAPD officials had developed ‘policing-the-police’ technologies – algorithms and monitoring devices designed to predict which officers were most likely to engage in violent conduct, then track the location of their vehicles.