Elements of Accountable AI

Six concerns that every organization developing or deploying AI should address

Image by Dall-E 3.

I’ve argued in previous posts that organizations deploying AI should focus on accountability. What exactly, though, does an Accountable AI effort cover?

AI raises many legal and ethical questions. Some of them are more relevant in some contexts than others. For example, the moral issues for autonomous weapons systems are somewhat different than those for auto-filling likely text in your email client. And some depend on the type of AI system. Intellectual property concerns over AI-generated outputs are only relevant for AI that can generate seemingly-novel creative works. Still, any good Accountable AI effort should consider the range of possible issues to determine which are relevant for any particular organization.

I see six major areas of concern for AI systems:

1. Correctness

2. Transparency

3. Fairness

4. Data Acquisition

5. Abuses

6. Societal Issues

Each has sub-categories, which are important to appreciate. The difference between bias (in the statistical sense) and discrimination (in the legal sense) is critically important to appreciate, but both are examples of fairness concerns.

Correctness

AI systems may get things wrong. In fact, sooner or later they are bound to. They are, after all, statistical engines for finding correlations across data, which imposes fundamental limits on their ability to encompass the complexity of the world or the mysteries of the human brain. And that can be a big problem. Zillow lost nearly $1 billion when its iBuyer algorithms led it to purchase too many houses at too-high prices.

The Zillow example illustrates why I use the term “correctness” rather than “accuracy.” Zillow’s system was accurate under certain conditions, but proved brittle as market conditions changed due to the Covid-19 pandemic, among other factors. Data scientists label that problem “robustness” of models. Another term used today is “alignment:” is the AI system pursuing the same objective the humans who designed it intended? While often discussed in terms of existential risk of AI superintelligences, alignment is a broader problem. AI systems pursue a mathematically-encoded “objective function,” but human goals are more nuanced and complex. A fourth sub-category of correctness might be called “appropriateness”. A customer service chatbot that outputs toxic and threatening speech might be giving an “accurate” response for the circumstances, but it’s not what the company wants.

A good conversation about AI correctness will include discussions of what happens when things go wrong. Problematic AI outputs open up risks, give rise to potential liability, and raise questions of who are responsible actors. When a car equipped with autonomous driving software crashes and kills someone in another car, for example, who is morally and legally at fault? What if a generative AI model goes wrong because a third party deliberately poisoned the training data?

Transparency

The second element of Accountable AI flows from the first. How can we know if an AI system is correct or not? Transparency is the broad category for understanding what AI systems are actually doing.

As with correctness, transparency has several sub-dimensions. One is awareness: whether those subject to AI decision-making even know that AI is involved. A second is interpretability, which is a technical term for models whose outputs can definitively be linked to inputs. Most AI models today are not interpretable in this formal sense, because that limits their power and creativity. However, it may be possible to reconstruct some aspects of the AI outputs, and to choose models and approaches that are more interpretable than others. A third dimension is explainability. Explainability means being able to answer “why” in some useful way, which is not exactly the same as interpretability. It might, for example, entail describing what data and features went into the model, or what factors are likely to have been significant.

Transparency is also connected to correctness in the other direction. A more-transparent model may well be poorer-performing, and less accurate. Considering and minimizing such tradeoffs are important aspect of Accountable AI practice.

Fairness

Sometimes an AI system produces results that are accurate, yet still ethically suspect. We don’t want AI systems that recommend men more highly for jobs simply because women have historically not performed as well at a company, due to sexism. And sometimes AI results are less accurate because of the effects of societal discrimination. Facial recognition systems that performed worse for women and people of color were the result of many factors, including fewer examples in training data, cameras optimized for performance on white faces, and standardized test datasets that were themselves biased.

I use the term “fairness” to emphasize the ethical questions at issue. We might call a system that harms disadvantaged groups “biased,” but in data science, the technical meaning of that term is any consistent pattern of inaccuracy. A facial recognition system that struggles with images featuring head coverings could be biased but not unfair… unless, as often the case, that functions as a proxy for something else we may be concerned about. (For example, Muslim women wearing a hijab.) “Discrimination” is the subset of biased or unfair actions subject to legal scrutiny. Those deploying AI should be concerned about all unfairness, but the requirements and consequences are different for illegal discrimination.

Data Acquisition

Machine learning systems, which means basically any modern implementation of AI, benefit from large amount of data. Generative AI models are typically “pretrained” (the “P” in ChatGPT) with particularly massive datasets, raising the stakes for sourcing that data. AI data acquisition raises at least four accountability concerns.

The first is privacy. Whether it’s predictive models used by social media platforms to analyze your prior activities and recommend products, or generative AI large language models that can “leak” private information sucked up in their training data, AI’s massive data collection requirements raise the stakes on the already serious concerns about data privacy. A separate issue is that whenever there are large stores of valuable data, we need to think about cybersecurity, which is also important for the AI models themselves. And for many AI systems, especially generative ones, there is a human supply chain involved in collecting and cleaning data, as well as fine-tuning models with human feedback. Those involved are often in the developing world, raising exploitation questions.

Finally, generative AI brings copyright issues to the fore. Many systems are trained on copyrighted material accessible on the internet. And some create outputs that strongly resemble copyrighted works. There are already significant copyright lawsuits involving both text and images. There will be similar questions around licensed proprietary business data, even when not technically subject to copyright.

Abuses

All of the prior categories involve AI being deployed for legitimate ends. The harmful results are an unintended consequence. There are, however, situations where AI is used intentionally to do harm. Two of the biggest areas are manipulation and misinformation. AI systems can be used to get people to act in ways contrary to their interests, or they can deliberately spread falsehoods. A major category of the latter involves deepfakes, which can involve images, video, or audio. Deepfakes are already being used to influence elections and commit crimes.

Many AI models are open source, raising concerns that even if their creators are well-intentioned and build in appropriate guardrails, other users may engage in dangerous activities.

Societal Issues

Finally, there are “big picture” problems that AI raises. These might not be significant for individual companies deploying generative AI solutions in their particular niche, but they are major concerns for the AI ecosystem as a whole.

One major example is job displacement. If AI systems take the place of human workers, it raises concerns both for the individuals experiencing lower pay or losing their employment. It also raises broad questions about economic inequality.

AI systems, especially generative models requiring massive computation for training, collectively use huge amounts of computing power, which means they are voracious consumers of water and electricity. AI has thus become part of the sustainability and ESG conversation. Along similar lines, the potential market concentration of AI developers adds to existing concerns about competition and antitrust. And government use of AI systems, especially biometric identification methods such as facial recognition, is becoming a big part of the global debate over surveillance. And finally, as AI becomes a major factor in economic development, business innovation, and military and intelligence activity, the geopolitics become significant.

What’s Missing?

You’ll notice some AI risks that aren’t listed here. I’m skeptical that we need to panic about existential threats of ultra-powerful AI systems. Even if one is concerned about alignment issues for artificial general intelligence, though, those are discussions focusing on the major generative AI model developers. I’m concentrating here on accountability concerns relevant to the much larger universe of organizations, private and governmental, who are engaged in building and deploying AI capabilities. Along similar lines, as noted earlier, there may be issues that are only relevant in certain industries or other contexts.

This is a list of challenges, not responses. A wealth of mechanisms such as audits, red teaming, and algorithmic debiasing come into play ones the issues are identified. Implementing solutions effectively is huge challenge. Understanding the problems, however, needs to be the first step.