The Beauty.AI case

Launched in 2016, the Beauty.AI tool was created to select “the First Beauty Queen or King Judged by Robots”, using age and facial recognition algorithms. Seven thousand people sent in their pictures through an app, but most of the 44 winners were white; only a handful were Asian, and only one had dark skin. This was despite the fact that, although the majority of contestants were white, many people of color submitted photos, including large groups from Africa and India. This was immediately considered a racist result, due to poor selection of the training dataset. The main problem was that the data the project used to establish standards of beauty were mainly composed by white people. Although the developers did not build the algorithm to treat light skin as a sign of beauty, the input data effectively led the robot judges to reach that conclusion.^[2]

The Amazon recruiting tool

In December 2018, Amazon scrapped its AI recruiting tool when the company discovered that the AI system showed bias against women. Amazon had been building computer programs since 2014 to review job applicants’ resumes, with the aim of mechanizing the search for top talent. The tool used AI to score job candidates from one to five stars. In 2015, however, Amazon discovered that the tool was not rating candidates for software developer jobs and other technical posts in a gender-neutral way. This was because Amazon’s computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.^[3]

In the past, loan applications from women were rejected more frequently than those from men, due to prejudice. In this case, any AI model trained on historical data is likely to reproduce the same pattern of discrimination. These issues can occur even if the training data does not contain any protected characteristics, such as gender or race. A variety of features in the training data are often closely correlated with protected characteristics (e.g. occupation, race, etc.). These ‘proxy variables’ enable the model to reproduce patterns of discrimination associated with those characteristics, even if its designers did not intend this.

These problems can occur in any statistical model. However, they are more likely to occur in AI systems because they can include a greater number of features, and may identify complex combinations of features that are proxies for protected characteristics. Many modern machine-learning methods are more powerful than traditional statistical approaches because they are better at uncovering non-linear patterns in high dimensional data. However, these also include patterns that reflect discrimination.^[4]

The US healthcare system uses commercial algorithms to guide health decisions. Obermeyer et al.^[5]found evidence of racial bias in one widely used algorithm, which meant that, among black and white patients assigned the same level of risk by the algorithm, the black patients were sicker than the white ones. The authors estimated that this racial bias reduced the number of black patients identified for extra care by more than half. Bias occurred because the algorithm used health costs as a proxy for health needs. Less money was spent on black patients with the same level of need as white patients, and the algorithm thus falsely concluded that black patients were healthier than equally sick white patients. In reality, the minor expenditure was caused by a number of racially biased factors, such as different access to treatment, levels of trust in the system, imbalances caused by healthcare givers, etc.