AI Safety Subproblems for Software Engineering Researchers

Works like (Yampolskiy, 2022) and (Russell, 2017) survey skepticism on the need for focused AI safety work. Debates are not new; In 1950 Turing (1950) argued against skeptics of thinking machines. Categories from (Yampolskiy, 2022) include:

1. (1)

   Priorities Objections: HLMI problems are too futuristic and so unimportant compared to other problems. This objection neglects evidence that timelines to HLMI are shortening, and that safety problems might take time to solve.

2. (2)

   Technical Objections: HLMI as unattainable. Recent successes contradict this position. Others mistakenly believe there will be trivial technical solutions to safety. Others view AI safety an impossible problem to work on currently (a view we hope diminish with work like this).

3. (3)

   “Biased Objections”: Safety considerations will slow progress or decrease funding. This has echos of 20th-century biased research into tobacco or pollutants (Oreskes and Conway, 2010). As AI advances, this is not sustainable.

Skepticism on advanced AI and existential risks is natural: these fears seem very weird and can feel like science fiction. Yet, significant evidence supports this weird future.

SE researchers have an active role to play in long-term AI safety, due to the nature of software. Transformational and risky capabilities come from when machines can form and execute complex plans. It seems likely that reasoning on code will be a key skill which enables these capabilities. Highlighting this importance, a first multi-million user deployment of billion+ parameter neural networks came not for NLP or CV or RL, but for SE with Github Copilot.

Traditional research problems directed at making complex software behave as expected can be adapted to apply to AI software and also machine-written software.

There are many survey papers on AI Safety (Everitt et al., 2018; Hendrycks et al., 2021; Juric et al., 2020; Sotala and Yampolskiy, 2014; Dwivedi et al., 2021; Russell et al., 2015; Critch and Krueger, 2020). In particular, (Everitt et al., 2018) provides a nice general survey. (Hendrycks et al., 2021) has a useful emphasis on ML. (Hendrycks and Mazeika, 2022) discusses assessing risks of AI work. (Tambon et al., 2021; Dey and Lee, 2021) are SE journal articles on ML safety (but are less HLMI focused).

AgiSafetyFundamentals.com collects a set of links and a course syllabus⁹⁹9https://agisafetyfundamentals.com/resources and ../ai-alignment-curriculum. Berkeley CHAI similarly provides a bibliography¹⁰¹⁰10https://humancompatible.ai/bibliography, ordered by topic and “priority”.

Books such as Bostrom (2017) collect key arguments in longer form. Other recent books include (Russell, 2019; Tegmark, 2017; Christian, 2021).

A nearer-term problem than HLMI is when a majority of source code is written by machines (at the abstraction level of current languages. Compiled and declarative high-level languages allow most code to be “machine-written” as per 1950s abstraction levels).

Not long after deploying a generative AI autocomplete system, Google engineers reported in 2022 that approximately 3% of new internal code was AI-written (Tabachnyk and Nikolov, 2022). On the path to 50% there are many safety-relevant problems.

In a world where most code is machine written, there needs to be ways to determine when the code might be misaligned with intent. A first step could accurately estimate the probability AI4SE outputs are correct. Work on calibrating ML probabilities includes (Lakshminarayanan et al., 2016; Guo et al., 2017; Kadavath et al., 2022). Careful attention is required: a claim like “the system says the code is 99% likely to be right, good enough” implies likely failure within 100s of tries. More complex modeling of uncertainty should account for varying degrees of negative impact (a 10% risk of over-counting the number of files in a directory is better than a 10% risk of deleting all the files in a directory). It could also help to localize certain regions of code which are uncertain or under-specified (eg, if asked to generate a button, the system might choose a button color or size even if it is not provided in the prompt specification). A related approach is to generate code with “holes” in uncertain regions (Guo et al., 2021). Learning about uncertainty might generalize to other generative AI systems with complex output (like in natural language (NL) or a robot action-space).

Humans might not be able to review all lines of code; faithful summaries can help developers understand machine-written software. Summarizing complex software outputs might also have lessons for summarizing other AI created plans that are too complex for easy review. Prior safety-motivated work has explored summarizing longform NL like books (Wu et al., 2021).

AI4SE has long studied automated summarization of source code (Zhang et al., 2022) or other artifacts like pull requests (Liu et al., 2019). The studied datasets are traditionally human written artifacts, but a long-term view likely should put increasing emphasis on artifacts coming from generators. While the summarizer might fail, its failures should ideally be anti-correlated with generator failures.

In the current SE paradigm, version control systems like Git makes each line of code traceable to a human author. As we move to a world where most code is machine written, code committed by a developer may not have been written by them; we thus need ways track what tool generated the code, under what conditions (e.g., prompting), and how it was audited. Commentary on this need has appeared previously (Bird et al., 2023).

While testing is an important enabler of software reliability today, gaps in testing are hopefully backstopped by having code written carefully by skilled and thoughtful humans. With improving code generators, we could expect that less human effort will be spent on implementation, and more on testing and verifying software. This conjecture is trackable with studies of developer time¹¹¹¹11eg, (Meyer et al., 2019) which estimated Microsoft developers spending 29% of time spent on implementation focused activities vs 19% on testing/reviewing/specification focused activities. If true we would expect time to shift away from implementation time..

Testing provides a formalized way to specify intent (Lahiri et al., 2022). Thus, a potential way to improve safety is to work on ways of identifying the most important and uncertain regions of code, and guiding developers to formalize their intents as tests.

Modern operating systems and run-times are responsible for assuring that software only accesses resources (like memory, compute time, files, cameras, etc) that it is permitted. This improves trust in software written by others. As the “others” shift from humans to AI systems, the need for such OS assurances will likely increase. Safety might be improved if an AI system could make specific, verifiable, claims, accompanying generated code (e.g, that it won’t touch files, or have certain runtime bounds; Necula’s proof-carrying code (Necula, 1997) is a historical example). Those claims could then be enforced at a system level. Improved systems controls can likely improve software safety that is intermixed with AI written code, and be a useful proxy when thinking about controlling AI outputs across domains.

There is interest in automated detection, prioritization, and repair of bugs and vulnerabilities (Russell et al., 2018; Le et al., 2022). This could have long-term benefits, but it’s important to note that vulnerability detection is a dual-use technology (vulnerabilities can be maliciously exploited). The security research community is usually already well aware of dual-use risks, but there must be expanded awareness in the SE community that AI can make exploiting vulnerabilities more scalable. Yet, it seems there is a reasonable long-term safety argument towards the research as hardening defenses and fixing vulnerabilities helps stop rapid changes from the status quo (e.g., an AI system hacking into cloud computing providers to acquire vastly more compute) without time for intervention, or help prevent the hacking of aligned AGI (Ladish and Heim, 2022).

In the 1950s, computers were large and very expensive, and operating them was a specialized occupation. Today, with PCs, smartphones and embedded devices, everyone is a ”computer operator”.

A natural conjecture is that on the path to HLMI, everyone will become a software engineer. Users will “speak apps into existence”, an extreme form of WYSIWYG (What You See Is What You Get) or no-code/low-code editors that let users click websites, documents, or apps into existence. This transformation is not just about autogenerated code, but everyone undertaking all parts of SE (problem specification, testing, deployment, etc). Changes here are difficult to track, but estimates of low-code usage can be proxies (Gartner, 2022).

There is a gap between the precision of a typical programming language and the ambiguity of natural language. Software creators of the future might not understand much about traditional programming languages. Challenges caused by this are apparent in existing low-code platforms with limited AI use (Rokis and Kirikova, 2022). One could imagine new programming languages or low-code paradigms that better mix together ambiguity and precision, and guide the user towards precision, particularly in areas with high risks of harms from intent ambiguity. While doing this, safety-conscious researchers could consider how their findings could generalize as systems scale to super-human capability or to other domains.

Effective techniques must be developed to educate non-technical users how automated SE tools can fail. This is a combination of the challenges of testing low-code (sadat Khorram et al., 2020), explaining AI decisions (Miller, 2017; Mohseni et al., 2021), and out-of-distribution detection (Yang et al., 2021). A safety-conscious researcher can also think about how these education techniques can be used to teach about failure modes of broader AI.

It seems beneficial to understand when the use of an AI4SE system should be discouraged; although this runs counter to the natural goals of improving productivity, it is essential for a responsible system. Work like (Cheng et al., 2022; Liao and Sundar, 2022) has studied trust of tools.

Software is an essential part of financial systems, health systems, and warfighting systems. As AI4SE becomes the norm, it will likely be used in writing software for the most critical parts of society.

If complex ML-based systems are deployed in critical areas, there are likely opportunities to apply SE techniques for testing and understanding ML systems. Existing CSafe in SE works are mostly in this SE4AI robustness area (Gerasimou et al., 2020; Baluta et al., 2020). Other SE4AI work has seeked to understand how to deploy AI systems (Martínez-Fernández et al., 2021; Gezici and Tarhan, 2022). Improved robustness and understanding can make aligned HLMI more stable.

Systems like ChatGPT are helping raise awareness that designing systems require normative decisions (Weidinger et al., 2021; Jakesch et al., 2023; Ganguli et al., 2023). There must be expanded understanding of similar failure modes and norms in AI4SE (for example, (Chen et al., 2021) explains how a code model might promote stereotypes of race or gender when writing algorithms or UIs). As machines author or edit software for the most important parts of society, the decisions become increasingly important. Additionally, proactive norms are needed in topics like self-modification of AI4SE tools or automated malware generation.

AI regulations are sometimes controversial (Zhang et al., 2021; Michael et al., 2022). However, when considering the use of AI4SE systems in the most safety-critical systems, the need for common enforceable rules becomes increasingly clear. Natural market forces encourage some safety precautions, but likely are not enough to prompt adequate investment into avoiding catastrophes from highly capable AI systems, while also avoiding “arms races” (Dafoe, 2018).

AI poses many complicated problems that are going to take significant effort. Thus, wide community involvement and a culture of safety and preparedness is needed.

Not all problems aid safety. We will discuss one example.

A natural direction of the field is to drive forward correctness metrics of tasks like automated code generation. However, capability drives are not necessarily a safety objective. For example, improving the fraction of methods passing tests in the HumanEval (Chen et al., 2021) dataset (which measure ability to synthesize programs) is not a safety objective. Similarly, optimizing user metrics like the fraction of generations accepted is not a safety objective. On the surface, one might argue that making AIs generate more correct code improves safety, compared to the alternative of AI writing incorrect code. Yet, we claim this is a bit of a false dichotomy. There is an alternative with the current status quo of human-authored code, which has many issues (high cost, inaccessible, error prone, etc), but gives more time to understand risks and safety problems. Pushing metrics like generation accuracy to their natural limit can lead to dangerously capable systems. Reaching the limit does not happen without incremental steps which on the surface seem harmless.

We are not advocating here the stronger view that progress on code tools or code generators should completely halt (there is huge upside to automating software development). Rather, we note that advances in correctness and capability, while important, don’t necessarily lead to greater long-term safety. Researchers, institutions, conferences, and the field should consider the portfolio of research being done. The proportion that is safety-focused should be brought in line with the amount of risk if the capabilities work on AI4SE succeeds and we reach the wild world where intelligent machines write most code in all parts of society.

We acknowledge that the distinction between this “anti-problem” and some of the other problems mentioned are not always ideally clear. Work on clearer boundaries and trade-offs is beneficial.