Heyn, «Berlin’s Wonderful Horse.»

Pfungst, Clever Hans.

«Clever Hans’ Again.»

Pfungst, Clever Hans.

Pfungst.

Lapuschkin et al., «Unmasking Clever Hans Predictors.»

See the work of philosopher Val Plumwood on the dualisms of intelligence-stupid, emotional-rational, and master-slave. Plumwood, «Politics of Reason.»

Turing, «Computing Machinery and Intelligence.»

Von Neumann, The Computer and the Brain, 44. This approach was deeply critiqued by Dreyfus, What Computers Can’t Do.

See Weizenbaum, «On the Impact of the Computer on Society,» After his death, Minsky was implicated in serious allegations related to convicted pedophile and rapist Jeffrey Epstein. Minsky was one of several scientists who met with Epstein and visited his island retreat where underage girls were forced to have sex with members of Epstein’s coterie. As scholar Meredith Broussard observes, this was part of a broader culture of exclusion that became endemic in AI: «As wonderfully creative as Minsky and his cohort were, they also solidified the culture of tech as a billionaire boys’ club. Math, physics, and the other ‘hard’ sciences have never been hospitable to women and people of color; tech followed this lead.» See Broussard, Artificial Unintelligence, 174.

Weizenbaum, Computer Power and Human Reason, 202–3.

Greenberger, Management and the Computer of the Future, 315.

Dreyfus, Alchemy and Artificial Intelligence.

Dreyfus, What Computers Can’t Do.

Ullman, Life in Code, 136–37.

See, as one of many examples, Poggio et al., «Why and When Can Deep – but Not Shallow – Networks Avoid the Curse of Dimensionality.»

Quoted in Gill, Artificial Intelligence for Society, 3.

Russell and Norvig, Artificial Intelligence, 30.

Daston, «Cloud Physiognomy.»

Didi-Huberman, Atlas, 5.

Didi-Huberman, 11.

Franklin and Swenarchuk, Ursula Franklin Reader, Prelude.

For an account of the practices of data colonization, see «Colonized by Data»; and Mbembé, Critique of Black Reason.

Fei-Fei Li quoted in Gershgorn, «Data That Transformed AI Research.»

Russell and Norvig, Artificial Intelligence, 1.

Bledsoe quoted in McCorduck, Machines Who Think, 136.

Mattern, Code and Clay, Data and Dirt, xxxiv-xxxv.

Ananny and Crawford, «Seeing without Knowing.»

Any list will always be an inadequate account of all the people and communities who have inspired and informed this work. I’m particularly grateful to these research communities: FATE (Fairness, Accountability, Transparency and Ethics) and the Social Media Collective at Microsoft Research, the AI Now Institute at NYU, the Foundations of AI working group at the École Normale Supérieure, and the Richard von Weizsäcker Visiting Fellows at the Robert Bosch Academy in Berlin.

Saville, «Towards Humble Geographies.»

For more on crowdworkers, see Gray and Suri, Ghost Work; and Roberts, Behind the Screen.

Canales, Tenth of a Second.

Zuboff, Age of Surveillance Capitalism.

Cetina, Epistemic Cultures, 3.

«Emotion Detection and Recognition (EDR) Market Size.»

Nelson, Tu, and Hines, «Introduction,» 5.

Danowski and de Castro, Ends of the World.

Franklin, Real World of Technology, 5.

Brechin, Imperial San Francisco.

Brechin, 29.

Agricola quoted in Brechin, 25.

Quoted in Brechin, 50.

Brechin, 69.

See, e. g., Davies and Young, Tales from the Dark Side of the City; and «Grey Goldmine.»

For more on the street-level changes in San Francisco, see Bloomfield, «History of the California Historical Society’s New Mission Street Neighborhood.»

«Street Homelessness.» See also «Counterpoints: An Atlas of Displacement and Resistance.»

Gee, «San Francisco or Mumbai?»

H. W. Turner published a detailed geological survey of the Silver Peak area in July 1909. In beautiful prose, Turner extolled the geological variety within what he described as «slopes of cream and pink tuffs, and little hillocks of a bright brick red.» Turner, «Contribution to the Geology of the Silver Peak Quadrangle, Nevada,» 228.

Lambert, «Breakdown of Raw Materials in Tesla’s Batteries and Possible Breaknecks.»

Bullis, «Lithium-Ion Battery.»

«Chinese Lithium Giant Agrees to Three-Year Pact to Supply Tesla.»

Wald, «Tesla Is a Battery Business.»

Scheyder, «Tesla Expects Global Shortage.»

Wade, «Tesla’s Electric Cars Aren’t as Green.»

Business Council for Sustainable Energy, «2019 Sustainable Energy in America Factbook.» U. S. Energy Information Administration, «What Is U. S. Electricity Generation by Energy Source?»

Whittaker et al., AI Now Report 2018.

Parikka, Geology of Media, vii – viii; McLuhan, Understanding Media.

Ely, «Life Expectancy of Electronics.»

Sandro Mezzadra and Brett Neilson use the term «extractivism» to name the relation between different forms of extractive operations in contemporary capitalism, which we see repeated in the context of the AI industry. Mezzadra and Neilson, «Multiple Frontiers of Extraction.»

Nassar et al., «Evaluating the Mineral Commodity Supply Risk of the US Manufacturing Sector.»

Mumford, Technics and Civilization, 74.

See, e. g., Ayogu and Lewis, «Conflict Minerals.»

Burke, «Congo Violence Fuels Fears of Return to 90s Bloodbath.»

«Congo ’s Bloody Coltan.»

«Congo ’s Bloody Coltan.»

«Transforming Intel’s Supply Chain with Real-Time Analytics.»

See, e. g., an open letter from seventy signatories that criticizes the limitations of the so-called conflict-free certification process: «An Open Letter.»

«Responsible Minerals Policy and Due Diligence.»

In The Elements of Power, David S. Abraham describes the invisible networks of rare metals traders in global electronics supply chains: «The network to get rare metals from the mine to your laptop travels through a murky network of traders, processors, and component manufacturers. Traders are the middlemen who do more than buy and sell rare metals: they help to regulate information and are the hidden link that helps in navigating the network between metals plants and the components in our laptops» [89].

«Responsible Minerals Sourcing.»

Liu, «Chinese Mining Dump.»

«Bayan Obo Deposit.»

Maughan, «Dystopian Lake Filled by the World’s Tech Lust.»

Hird, «Waste, Landfills, and an Environmental Ethics of Vulnerability,» 105.

Abraham, Elements of Power, 175.

Abraham, 176.

Simpson, «Deadly Tin Inside Your Smartphone.»

Hodal, «Death Metal.»

Hodal.

Tully, «Victorian Ecological Disaster.»

Starosielski, Undersea Network, 34.

See Couldry and Mejías, Costs of Connection, 46.

Couldry and Mejías, 574.

For a superb account of the history of undersea cables, see Starosielski, Undersea Network.

Dryer, «Designing Certainty,» 45.

Dryer, 46.

Dryer, 266-68.

More people are now drawing attention to this problem – including researchers at AI Now. See Dobbe and Whittaker, «AI and Climate Change.»

See, as an example of early scholarship in this area, Ensmenger, «Computation, Materiality, and the Global Environment.»

Hu, Prehistory of the Cloud, 146.

Jones, «How to Stop Data Centres from Gobbling Up the World’s Electricity.» Some progress has been made toward mitigating these concerns through greater energy efficiency practices, but significant long-term challenges remain. Masanet et al., «Recalibrating Global Data Center Energy – Use Estimates.»

Belkhir and Elmeligi, «Assessing ICT Global Emissions Footprint»; Andrae and Edler, «On Global Electricity Usage.»

Strubell, Ganesh, and McCallum, «Energy and Policy Considerations for Deep Learning in NLP.»

Strubell, Ganesh, and McCallum.

Sutton, «Bitter Lesson.»

«AI and Compute.»

Cook et al., Clicking Clean.

Ghaffary, «More Than 1,000 Google Employees Signed a Letter.» See also «Apple Commits to Be 100 Percent Carbon Neutral»; Harrabin, «Google Says Its Carbon Footprint Is Now Zero»; Smith, «Microsoft Will Be Carbon Negative by 2030.»

«Powering the Cloud.»

«Powering the Cloud.»

«Powering the Cloud.»

Hogan, «Data Flows and Water Woes.»

«Off Now.»

Carlisle, «Shutting Off NSA’s Water Gains Support.»

Materiality is a complex concept, and there is a lengthy literature that contends with it in such fields as STS, anthropology, and media studies. In one sense, materiality refers to what Leah Lievrouw describes as «the physical character and existence of objects and artifacts that makes them useful and usable for certain purposes under particular conditions.» Lievrouw quoted in Gillespie, Boczkowski, and Foot, Media Technologies, 25. But as Diana Coole and Samantha Frost write, «Materiality is always something more than ‘mere’ matter: an excess, force, vitality, relationality, or difference that renders matter active, self-creative, productive, unproductive.» Coole and Frost, New Materialisms, 9.

United Nations Conference on Trade and Development, Review of Maritime Transport, 2017.

George, Ninety Percent of Everything, 4.

Schlanger, «If Shipping Were a Country.»

Vidal, «Health Risks of Shipping Pollution.»

«Containers Lost at Sea–2017 Update.»

Adams, «Lost at Sea.»

Mumford, Myth of the Machine.

Labban, «Deterritorializing Extraction.» For an expansion on this idea, see Arboleda, Planetary Mine.

Ananny and Crawford, «Seeing without Knowing.»

Wilson, «Amazon and Target Race.»

Lingel and Crawford, «Alexa, Tell Me about Your Mother.»

Federici, Wages against Housework; Gregg, Counterproductive.

In The Utopia of Rules, David Graeber details the sense of loss experienced by white-collar workers who now have to enter data into the decision-making systems that have replaced specialist administrative support staff in most professional workplaces.

Smith, Wealth of Nations, 4–5.

Marx and Engels, Marx-Engels Reader, 479. Marx expanded on this notion of the worker as an «appendage» in Capital, vol. 1: «In handicrafts and manufacture, the worker makes use of a tool; in the factory, the machine makes use of him. There the movements of the instrument of labor proceed from him, here it is the movements of the machine that he must follow. In manufacture the workers are parts of a living mechanism. In the factory we have a lifeless mechanism which is independent of the workers, who are incorporated into it as its living appendages.» Marx, Das Kapital, 548–49.

Luxemburg, «Practical Economies,» 444.

Thompson, «Time, Work-Discipline, and Industrial Capitalism.»

Thompson, 88–90.

Werrett, «Potemkin and the Panopticon,» 6.

See, e. g., Cooper, «Portsmouth System of Manufacture.»

Foucault, Discipline and Punish; Horne and Maly, Inspection House.

Mirzoeff, Right to Look, 58.

Mirzoeff, 55.

Mirzoeff, 56.

Gray and Suri, Ghost Work.

Irani, «Hidden Faces of Automation.»

Yuan, «How Cheap Labor Drives China’s A. I. Ambitions»; Gray and Suri, «Humans Working behind the AI Curtain.»

Berg et al., Digital Labour Platforms.

Roberts, Behind the Screen; Gillespie, Custodians of the Internet, 111–40.

Silberman et al., «Responsible Research with Crowds.»

Silberman et al.

Huet, «Humans Hiding behind the Chatbots.»

Huet.

See Sadowski, «Potemkin AI.»

Taylor, «Automation Charade.»

Taylor.

Gray and Suri, Ghost Work.

Standage, Turk, 23.

Standage, 23.

See, e. g., Aytes, «Return of the Crowds,» 80.

Irani, «Difference and Dependence among Digital Workers,» 225.

Pontin, «Artificial Intelligence.»

Menabrea and Lovelace, «Sketch of the Analytical Engine.»

Babbage, On the Economy of Machinery and Manufactures, 39–43.

Babbage evidently acquired an interest in quality-control processes while trying (vainly) to establish a reliable supply chain for the components of his calculating engines.

Schaffer, «Babbage’s Calculating Engines and the Factory System,» 280.

Taylor, People’s Platform, 42.

Katz and Krueger, «Rise and Nature of Alternative Work Arrangements.»

Rehmann, «Taylorism and Fordism in the Stockyards,» 26.

Braverman, Labor and Monopoly Capital, 56, 67; Specht, Red Meat Republic.

Taylor, Principles of Scientific Management.

Marx, Poverty of Philosophy, 22.

Qiu, Gregg, and Crawford, «Circuits of Labour»; Qiu, Goodbye iSlave.

Markoff, «Skilled Work, without the Worker.»

Guendelsberger, On the Clock, 22.

Greenhouse, «McDonald’s Workers File Wage Suits.»

Greenhouse.

Mayhew and Quinlan, «Fordism in the Fast Food Industry.»

Ajunwa, Crawford, and Schultz, «Limitless Worker Surveillance.»

Mikel, «WeWork Just Made a Disturbing Acquisition.»

Mahdawi, «Domino’s ‘Pizza Checker’ Is Just the Beginning.»

Wajcman, «How Silicon Valley Sets Time.»

Wajcman, 1277.

Gora, Herzog, and Tripathi, «Clock Synchronization.»

Eglash, «Broken Metaphor,» 361.

Kemeny and Kurtz, «Dartmouth Timesharing,» 223.

Eglash, «Broken Metaphor,» 364.

Brewer, «Spanner, TrueTime.»

Corbett et al., «Spanner,» 14, cited in House, «Synchronizing Uncertainty,» 124.

Galison, Einstein’s Clocks, Poincaré’s Maps, 104.

Galison, 112.

Colligan and Linley, «Media, Technology, and Literature,» 246.

Carey, «Technology and Ideology.»

Carey, 13.

This contrasts with what Foucault called the «microphysics of power» to describe how institutions and apparatuses create particular logics and forms of validity. Foucault, Discipline and Punish, 26.

Spargo, Syndicalism, Industrial Unionism, and Socialism.

Personal conversation with the author at an Amazon fulfillment center tour, Robbinsville, N.J., October 8, 2019.

Muse, «Organizing Tech.»

Abdi Muse, personal conversation with the author, October 2, 2019.

Gurley, «60 Amazon Workers Walked Out.»

Muse quoted in Organizing Tech.

Desai quoted in Organizing Tech.

Estreicher and Owens, «Labor Board Wrongly Rejects Employee Access to Company Email.»

This observation comes from conversations with various labor organizers, tech workers, and researchers, including Astra Taylor, Dan Greene, Bo Daley, and Meredith Whittaker.

Kerr, «Tech Workers Protest in SF.»

National Institute of Standards and Technology (NIST), «Special Database 32-Multiple Encounter Dataset (MEDS).»

Russell, Open Standards and the Digital Age.

Researchers at NIST (then the National Bureau of Standards, NBS) began working on the first version of the FBI’s Automated Fingerprint Identification System in the late 1960s. See Garris and Wilson, «NIST Biometrics Evaluations and Developments,» 1.

Garris and Wilson, 1.

Garris and Wilson, 12.

Sekula, «Body and the Archive,» 7.

Sekula, 18–19.

Sekula, 17.

See, e. g., Grother et al., «2017 IARPA Face Recognition Prize Challenge (FRPC).»

See, e. g., Ever AI, «Ever AI Leads All US Companies.»

Founds et al., «NIST Special Database 32.»

Curry et al., «NIST Special Database 32 Multiple Encounter Dataset I (MEDS-I),» 8.

See, e. g., Jaton, «We Get the Algorithms of Our Ground Truths.»

Nilsson, Quest for Artificial Intelligence, 398.

«ImageNet Large Scale Visual Recognition Competition (ILSVRC).»

In the late 1970s, Ryszard Michalski wrote an algorithm based on symbolic variables and logical rules. This language was popular in the 1980s and 1990s, but as the rules of decision-making and qualification became more complex, the language became less usable. At the same moment, the potential of using large training sets triggered a shift from this conceptual clustering to contemporary machine learning approaches. Michalski, «Pattern Recognition as Rule-Guided Inductive Inference.»

Bush, «As We May Think.»

Light, «When Computers Were Women»; Hicks, Programmed Inequality.

As described in Russell and Norvig, Artificial Intelligence, 546.

Li, «Divination Engines,» 143.

Li, 144.

Brown and Mercer, «Oh, Yes, Everything’s Right on Schedule, Fred.»

Lem, «First Sally (A), or Trurl’s Electronic Bard,» 199.

Lem, 199.

Brown and Mercer, «Oh, Yes, Everything’s Right on Schedule, Fred.»

Marcus, Marcinkiewicz, and Santorini, «Building a Large Annotated Corpus of English.»

Klimt and Yang, «Enron Corpus.»

Wood, Massey, and Brownell, «FERC Order Directing Release of Information,» 12.

Heller, «What the Enron Emails Say about Us.»

Baker et al., «Research Developments and Directions in Speech Recognition.»

I have participated in early work to address this gap. See, e. g., Gebru et al., «Datasheets for Datasets.» Other researchers have also sought to address this problem for AI models; see Mitchell et al., «Model Cards for Model Reporting»; Raji and Buolamwini, «Actionable Auditing.»

Phillips, Rauss, and Der, «FERET (Face Recognition Technology) Recognition Algorithm Development and Test Results,» 9.

Phillips, Rauss, and Der, 61.

Phillips, Rauss, and Der, 12.

See Aslam, «Facebook by the Numbers (2019)»; and «Advertising on Twitter.»

Fei-Fei Li, as quoted in Gershgorn, «Data That Transformed AI Research.»

Deng et al., «ImageNet.»

Gershgorn, «Data That Transformed AI Research.»

Gershgorn.

Markoff, «Seeking a Better Way to Find Web Images.»

Hernandez, «CU Colorado Springs Students Secretly Photographed.»

Zhang et al., «Multi-Target, Multi-Camera Tracking by Hierarchical Clustering.»

Sheridan, «Duke Study Recorded Thousands of Students’ Faces.»

Harvey and LaPlace, «Brainwash Dataset.»

Locker, «Microsoft, Duke, and Stanford Quietly Delete Databases.»

Murgia and Harlow, «Who’s Using Your Face?» When the Financial Times exposed the contents of this dataset, Microsoft removed the set from the internet, and a spokesperson for Microsoft claimed simply that it was removed «because the research challenge is over.» Locker, «Microsoft, Duke, and Stanford Quietly Delete Databases.»

Franceschi-Bicchierai, «Redditor Cracks Anonymous Data Trove.»

Tockar, «Riding with the Stars.»

Crawford and Schultz, «Big Data and Due Process.»

Franceschi-Bicchierai, «Redditor Cracks Anonymous Data Trove.»

Nilsson, Quest for Artificial Intelligence, 495.

And, as Geoff Bowker famously reminds us, «Raw data is both an oxymoron and a bad idea; to the contrary, data should be cooked with care.» Bowker, Memory Practices in the Sciences, 184-85.

Fourcade and Healy, «Seeing Like a Market,» 13, emphasis added.

Meyer and Jepperson, «‘Actors’ of Modern Society.»

Gitelman, «Raw Data» Is an Oxymoron, 3.

Many scholars have looked closely at the work these metaphors do. Media studies professors Cornelius Puschmann and Jean Burgess analyzed the common data metaphors and noted two widespread categories: data «as a natural force to be controlled and [data] as a resource to be consumed.» Puschmann and Burgess, «Big Data, Big Questions,» abstract. Researchers Tim Hwang and Karen Levy suggest that describing data as «the new oil» carries connotations of being costly to acquire but also suggests the possibility of «big payoffs for those with the means to extract it.» Hwang and Levy, «‘The Cloud’ and Other Dangerous Metaphors.»

Stark and Hoffmann, «Data Is the New What?»

Media scholars Nick Couldry and Ulises Mejías call this «data colonialism,» which is steeped in the historical, predatory practices of colonialism but married to (and obscured by) contemporary computing methods. However, as other scholars have shown, this terminology is double-edged because it can occlude the real and ongoing harms of colonialism. Couldry and Mejías, «Data Colonialism»; Couldry and Mejías, Costs of Connection; Segura and Waisbord, «Between Data Capitalism and Data Citizenship.»

They refer to this form of capital as «ubercapital.» Fourcade and Healy, «Seeing Like a Market,» 19.

Sadowski, «When Data Is Capital,» 8.

Sadowski, 9.

Here I’m drawing from a history of human subjects review and largescale data studies coauthored with Jake Metcalf. See Metcalf and Crawford, «Where Are Human Subjects in Big Data Research?»

«Federal Policy for the Protection of Human Subjects.»

See Metcalf and Crawford, «Where Are Human Subjects in Big Data Research?»

Seo et al., «Partially Generative Neural Networks.» Jeffrey Brantingham, one of the authors, is also a co-founder of the controversial predictive policing company PredPol. See Winston and Burrington, «A Pioneer in Predictive Policing.»

«CalGang Criminal Intelligence System.»

Libby, «Scathing Audit Bolsters Critics’ Fears.»

Hutson, «Artificial Intelligence Could Identify Gang Crimes.»

Hoffmann, «Data Violence and How Bad Engineering Choices Can Damage Society.»

Weizenbaum, Computer Power and Human Reason, 266.

Weizenbaum, 275-76.

Weizenbaum, 276.

For more on the history of extraction of data and insights from marginalized communities, see Costanza-Chock, Design Justice; and D’Ignazio and Klein, Data Feminism.

Revell, «Google DeepMind’s NHS Data Deal ‘Failed to Comply.’»

«Royal Free-Google DeepMind Trial Failed to Comply.»

Fabian, Skull Collectors.

Gould, Mismeasure of Man, 83.

Kolbert, «There’s No Scientific Basis for Race.»

Keel, «Religion, Polygenism and the Early Science of Human Origins.»

Thomas, Skull Wars.

Thomas, 85.

Kendi, «History of Race and Racism in America.»

Gould, Mismeasure of Man, 88.

Mitchell, «Fault in His Seeds.»

Horowitz, «Why Brain Size Doesn’t Correlate with Intelligence.»

Mitchell, «Fault in His Seeds.»

Gould, Mismeasure of Man, 58.

West, «Genealogy of Modern Racism,» 91.

Bouche and Rivard, «America ’s Hidden History.»

Bowker and Star, Sorting Things Out, 319.

Bowker and Star, 319.

Nedlund, «Apple Card Is Accused of Gender Bias»; Angwin et al., «Machine Bias»; Angwin et al., «Dozens of Companies Are Using Facebook to Exclude.»

Dougherty, «Google Photos Mistakenly Labels Black People ‘Gorillas’»; Perez, «Microsoft Silences Its New A. I. Bot Tay»; McMillan, «It’s Not You, It’s It»; Sloane, «Online Ads for High-Paying Jobs Are Targeting Men More Than Women.»

See Benjamin, Race after Technology; and Noble, Algorithms of Oppression.

Greene, «Science May Have Cured Biased AI»; Natarajan, «Amazon and NSF Collaborate to Accelerate Fairness in AI Research.»