Prof. Jackie Leach Scully1, Dr Georgia van Toorn2, Dr Sandra Gendera3
1Disability Innovation Institute, UNSW, Sydney, Australia, 2School of Social Sciences, UNSW, Sydney, Australia, 3Social Policy Research Centre, UNSW, Sydney, Australia
Biography:
Jackie Leach Scully is Professor of Bioethics and Director of the Disability Innovation Institute at UNSW.
Abstract:
This presentation considers the ethical implications of failed recognition by artificial intelligence and associated technologies, such as automated decision-making (ADM). These technologies raise numerous ethical issues, including questions of accountability and transparency, and more recently issues of bias and discrimination in the collection and handling of data.
We have been engaged in an empirically focused study of the place of ADM systems in the lives of people with disability in Australia. Our disabled informants have identified several situations in which they felt the datafication of AI is unable to represent adequately the realities of embodied difference and social experience, including disability. One point often mentioned, but rarely discussed in the AI ethics literature, is automated systems’ failures of recognition.
Using the method of dialogue groups, our participants discussed fictional scenarios in which ADM was used and related them to their own experience with health and social care. They highlighted three distinct forms of failed or misrecognition: (i) of disabled experience; (ii) of disabled knowledge; and (iii) of the ontological presence of the disabled person. We discuss here the consequences of misrecognition for people with disability in the healthcare and other social support systems, where the use of ADM and AI is increasing. We also consider whether AI misrecognition is a problem specific to disabled people, or whether it affects socially and economically marginalised populations in general.