A/HRC/56/68 2. Health care (a) Health risk scores 40. Artificial intelligence can be utilized to create health risk scores for a variety of purposes in health care, including medical diagnosis and care planning. Racially discriminatory effects can occur when such algorithms are used to allocate health-care resources, because of algorithmic design and the data used to train artificial intelligence systems. There are cases in which such algorithms have been designed to identify who should qualify for extra care, using previous health-care costs as a proxy for medical needs. The data on which such decisions are based can be influenced by previous lack of adequate access to health care among those from marginalized racial and ethnic groups in the context of systemic racism, as well as different disease patterns due to disparities in the socioeconomic determinants of health. 41. In the United States, a calculator was developed to assist health-care providers in assessing the likelihood of successful vaginal birth after caesarean delivery. A study in 2019 revealed bias in the calculator’s foundational algorithm. The calculator had two race-based correction factors, which resulted in lower predicted vaginal birth success rates for women of African descent and Hispanic women compared with white women with similar characteristics. Because of these correction factors, the calculator potentially worsened racial disparities in maternal health outcomes by discouraging clinicians from offering vaginal birth to women of African descent and Hispanic women, leading to higher rates of caesarean sections.62 (b) Disease detection 42. Another application of artificial intelligence technologies is disease detection, including cancer detection.63 Artificial intelligence systems trained on extensive data sets comprising thousands or millions of images, including radiological scans, pathology images and photographs, can learn to distinguish between normal and cancerous lesions. 64 This deployment of artificial intelligence can significantly aid in early cancer detection, potentially saving lives while improving health-care system efficiency. However, those from marginalized racial and ethnic groups may not benefit equally from such advancements due to the algorithms not being generalizable to patient populations that are not adequately represented in the training data. Researchers have suggested that the use of artificial intelligence algorithms for skin cancer detection shows poorer performance for individuals with darker skin tones because many of the publicly available image data sets used to train them are biased, with a lack of diversity in skin tones and ethnic backgrounds.65 For example, a survey of 21 open-access skin lesion data sets, containing over 100,000 images, revealed a significant underrepresentation of images of darker skin.66 (c) Artificial intelligence-enabled medical devices 43. Artificial intelligence is being developed and utilized alongside other developments in health-care technologies, including health-care devices. Many of these devices are artificial intelligence-enabled, and racial bias can become embedded in their operation. For example, in the United Kingdom, a report showed that bias was baked into the operation of medical devices at all stages of their development, including during phases involving algorithm development and machine learning. During the coronavirus disease (COVID-19) 62 63 64 65 66 GE.24-08849 Darshali A. Vyas and others, “Challenging the use of race in the Vaginal Birth after Cesarean Section Calculator”, Women’s Health Issues, vol. 29, No. 3 (2019). Privacy International submission. Likhitha Kolla and Ravi B. Parikh, “Uses and limitations of artificial intelligence for oncology”, Cancer, 30 March 2024. David Wen and others, “Characteristics of publicly available skin cancer image datasets: a systematic review”, The Lancet Digital Health, vol. 4, No. 1 (2022). Ibid. See also Privacy International submission. 11