Perspectives on the Social Impacts of
Reinforcement Learning with Human Feedback

As RLHF is gaining rapid traction, now is the ideal time to consider its potential impacts on society ME2 . The transformational potential of RLHF makes it critical to consider how broadened application of RLHF-based technologies may impact various stakeholders, what ethical concerns might arise as a result, how it may affect social and ethical challenges, and how governance may be utilized to mitigate risks. This analysis is timely and justified considering the current state of AI safety research. According to a 2022 report by the Center for Security and Emerging Technology, research in the top three areas of AI safety—robustness, interpretability, and reward learning—have seen explosive growth in the past decade CSET4 . Reward learning is critically concerned with reducing the risk of disparity between intended and observed outcomes, yet work in the area is less developed relative to robustness and interpretability research. This discrepancy extends to the study of related social and ethical implications CSET4 . This report therefore seeks to fill this gap and step toward expanding responsible discussion of reward learning methods like RLHF.

The objective of this report is threefold: first, to provide a systematic study of the social effects of RLHF; second, to identify key social and ethical issues of RLHF; and third, to discuss social impacts for stakeholders. We propose that continued development of RLHF has a net positive social impact and is thus worth continued pursuit. We define an impact to be any direct or indirect effect that “includes both positive and negative results,” and a benefit to be “a positive impact that produces a good result” HELP1 . We assume social impact to be the net effect on relevant stakeholders, which may include individuals, families, communities, organizations, nations, or global societies as a whole HELP1 . Social impacts identified in this report are evaluated relative to the social baseline—the environment in the absence of the technology in question HELP1 ; SIA .

As an effective alignment technique, RLHF has significant potential to assist in mitigating harmful content generation that results from large language models (LLMs) and improve information integrity¹¹1Information integrity refers to the dependability and trustworthiness of information FOOTNOTE .. LLM deficiencies are well-documented and range from biased outputs to leaked private data to misinformation and adversarial attack GML1 . Current approaches to moderating LLMs are cumbersome, require more data, or are overly complex GML1 . RLHF is a method that promises improved truthfulness and reduced toxicity of LLMs without significantly compromising performance or creating issues such as reduced representation of minorities in textual output GML1 . For instance, InstructGPT—trained with RLHF—exhibits enhanced ability versus GPT-3 to generate truthful and informative responses and follow unfamiliar instructions (Figure 1) GML1 . Combined with development of limitations on harmful output production, RLHF has great potential toward generation of positive content for assistive technologies, information sharing, and recommender/advice systems.

Even so, work remains to be done in order to improve the reliability of RLHF-based models. RLHF technologies like ChatGPT can still suffer from inappropriate and harmful outputs upon user request. The creators of ChatGPT and InstructGPT themselves described these technologies as perhaps being too obedient to user instruction GML1 . Such issues may be approached by combining RLHF with steerability methods or by modifying sampling procedures during training. It may also be useful to incorporate aspects of AI explainability, giving RLHF agents the ability to decline to comply with harmful requests and explain why they have done so GML2 ; morals . Overall, what renders an output harmful often depends on context, and this can complicate model design GML1 .

Despite acting in broad alignment with human values, RLHF can still be misused for misinformation, oppression, or perpetuation of societal prejudice if guardrails are not established ME1 . RLHF-based content generation—whether visual, textual, auditory, or other forms of media—has application in disinformation and automated trolling, which can lead to compromised election integrity and public distrust in media and undermine the very fabric of organized governance in society. One can see that the power of RLHF in the disinformation context is threefold. First, human-machine teaming with RLHF models can accelerate message iteration, effectively increasing the productive power of disinformation campaigns CSET1 . Second, greater understanding of human behavior, preferences, and value systems could aid reconnaissance, leading to better mimicking of human activity, viewpoint manipulation, targeted messaging, conspiracy narrative generation, advancement of political narratives, and identification of feeble social fissures CSET5 . Third, knowledge that AI holds such potential may further erode trust and accelerate descent into the cynicism that advances disinformation CSET5 .

“A people that no longer can believe anything cannot make up its mind. It is deprived not only of its capacity to act but also of its capacity to think and to judge. And with such a people you can then do what you please.” —Hannah Arendt HA

There are a number of steps we can take to counter misuse of RLHF. To start, disinformation campaigns bear an innate limitation on their scale and scope. What makes disinformation effective? Beyond content generation, a successful effort at disinformation is heavily dependent on administration. While automation may free more humans to work on these tasks, propagation of content requires financial and technical infrastructure CSET1 . Perhaps the best mitigation for misuse of RLHF is to address governance of such infrastructure CSET1 . Notably, those who control such infrastructure may wield disproportionate power over the direction of RLHF applications.

More broadly, methods to counter AI disinformation will likely be equally useful for RLHF. Cooperation and intelligence sharing between governments and industry parties will be key to developing early warning systems for disinformation campaigns, enabling rapid response, threat information sharing, and cross-platform defense CSET1 . Since openly released research always carries the potential for misuse, AI researchers must develop more formalized guidelines for guarding against misuse and recommending mitigations. There must be a process by which media outlets can report on disinformation without amplifying its effects. Finally, public resistance to ML-enabled disinformation must be boosted by improving media literacy and increasing the accuracy of public conceptions of AI CSET1 ; CSET5 ; ME6 . At present, “A good deal of fear and concern about uncontrollable AI is now being displayed in public discourse,” which has led to confusion regarding autonomy and the critical role of humans throughout the AI development and deployment pipeline ME6 . Researchers must be transparent and understandable in how their work is communicated to the public, and media outlets must avoid misleading or over-sensationalized journalism about AI. At the same time, the risk of manipulation can be addressed by increasing digital literacy to enhance personal autonomy and awareness among the general public.

A core goal of AI research is to produce systems that behave in ways consistent with human values and intentions. There exist varying definitions of AI alignment with human intentions. Askell et al. (2021) define a well-aligned AI to be one that is “helpful, honest, and harmless,” with the understanding that “these are subtle and ambiguous criteria, and the best AI behavior will involve a compromise between them” ME1 . Current ML approaches tend to suffer from misalignment between the objectives of resulting systems and human values RLHF9 . Even if we were to observe model behavior fully consistent with human preferences (outer alignment), it is difficult to guarantee true inner alignment²²2In the context of AI safety, an inner alignment failure refers to any situation in which an AI agent optimizes for goals or objectives different from those we have asked of it INTERNALALIGNMENT . without ulterior motives RLHF2 ; INTERNALALIGNMENT .

RLHF is an important step forward in aligning AI systems with human values as it provides more nuanced guidance than traditional ML and RL, which struggle to capture the full extent of human preference. GML1 . Specifically, RL algorithms learn the highest-reward path toward a stated objective, sometimes involving actions that lend to economic or physical harm ME2 . In contrast, there is evidence that RLHF trains models to act in accordance with both explicit (following instructions) and implicit (staying truthful, unbiased, and unhurtful) intentions GML1 . More broadly, RLHF is an important vantage point for exploring improved systems of value alignment. Insights gained through RLHF are likely transferable to other alignment methods RLHF7 . Even if RLHF does not completely resolve concerns over inner alignment, the failures it identifies and knowledge it confers to reward and policy modeling are applicable to enhancing safety, reliability, and trustworthiness of AI in social and collaborative situations RLHF7 .

As AI becomes democratized, how do we construct systems that are sensitive to a diversity of perspectives and value systems and properly aligned in such contexts? Can we design a unified values framework, or should value alignment be restricted to cultural-specific contexts, much like law enforcement differences between nations? Could RLHF make inter- and intra-regional differences in conceptions of morality and ethics more salient? A significant factor in the net impact of RLHF models lies in to whom such models are aligned GML1 . Challenges exist in designing an alignment process that is fair, unbiased, and transparent while also bearing suitable accountability mechanisms GML1 . This is relevant in light of unresolved questions over how fundamentally conflicting feedback, values, and preferences should be reconciled, and the fact that there is no consensus across society on any single unified moral theory. Gabriel (2020) suggests pursuing a principle-based approach, whereby models are built to reflect fair principles endorsed by all despite variation in moral beliefs GABRIEL . Another path forward concerns training models that are aligned with general principles and preferences, with use of subsequent fine-tuning to condition models to preferences of specific groups. Additional issues are raised by the fact that developer choices can unintentionally impact the behavior of RLHF methods ME4 . It is perhaps more useful to develop RLHF under assumptions of moral uncertainty, which supposes for any decision that one’s motives are driven by several plausible ethical theories ME5 . Further consideration must be given to the broad question: should AI agents be able to exhibit the myriad moral and ethical convictions espoused by humans?

With proper deployment, RLHF can reduce bias at multiple levels in the AI production pipeline. Broadly speaking, AI is affected at multiple levels of development by historical bias which affects data generation, representation bias which affects sampling and population studies, measurement bias due to inaccurate data stemming and structural discrimination against groups, aggregation bias due to over-reliance on one-size-fits all models, learning and evaluation bias during model training, and deployment bias due to disparity between intended and observed application CS1 . Preliminary analysis of RLHF results suggests it can be leveraged to mitigate long-standing effects of historical, representation, and measurement bias by balancing human feedback with representation and expertise across a diverse range of human annotators morals ; ME4 . RLHF is not unsusceptible to bias or misuse, but it leverages human feedback to counter algorithmic bias directly and efficiently in comparison to existing approaches GML1 ; morals . In this light, RLHF is an important tool not only for its potential to transform AI capabilities, but also toward combating systemic inequality issues perpetuated by algorithmic development.

By reducing computational cost, RLHF can open the door to democratization of AI technologies to all levels of society regardless of level of development. In particular, RLHF yields smaller models requiring less compute to achieve state-of-the-art performance GML1 , which is critical for building practical AI technologies that are deployable across the world and especially to lower-income areas and developing nations. The reduced need for training data can mitigate concerns around data scraping and privacy, security, and surveillance, all of which are issues involved in traditional ML GML1 . Data collection often disproportionately impacts vulnerable groups in negative ways: data may be misused by technology companies and governments to, for example, track immigrants, and instances of surveillance used to solidify systemic discrimination against subpopulations are well-documented ME2 . RLHF thus makes it easier to achieve better outcomes without significantly compromising privacy.

RLHF has potential to transform how we reconcile cross-cultural perspectives and approach peaceful dialogue. Cross-cultural feedback is critical to ensuring technology is deployable in contexts beyond domestic production. By soliciting human feedback that encompasses a diversity of viewpoints and cultural norms, RLHF technologies can be culturally aware and usable beyond narrow, culture-specific settings. Even minor cultural awareness can facilitate communication in a number of contexts. A salient example is in education. Mitigating stress associated with feedback interactions in learning is critical to supporting student education BIAS1 . Yet studies have shown that cross-cultural feedback conversations between teachers and students can compound stress and lead to reduced learning (e.g., via decreased ability to ask questions, “absorb information, and develop professional and mentoring relationships”), worsened long-term education outcomes, and increased cognitive load for teachers if approached incorrectly BIAS1 . This was further exacerbated for interactions between teachers of well-represented identities and students from underrepresented groups BIAS1 . In this context, RLHF technologies can help overcome such difficulties, whether by moderating conversation or suggesting appropriate ways to approach cross-cultural communication. This benefit extends beyond education into sectors such as customer service and entertainment.

By allowing AI agents to learn from human expertise, RLHF can facilitate development of more adaptable AI systems for use in various industries RLHF8 . Potential applications of RLHF include enhanced resource management, customer service, online education, eldercare, and clinical decision support EXAMPLES . Adaptive recommendations could better account for personal and cultural preferences and human intentions; value-aligned technologies could better accommodate individual preferences regarding communication, mobility, and living habits; human-guided diagnostics could improve clarity in decision-making. RLHF can better foster trust with users in order to boost business outcomes across industries and accelerate technology adoption to improve efficiency and economic output.

Concurrently, RLHF can possibly heighten big-tech’s advantage and hasten progress towards dangerous AI capabilities. Notable RLHF advances have been achieved by well-financed research laboratories and big-tech companies such as OpenAI and DeepMind, which can afford to spend enormous amounts of money on creating large datasets for RLHF algorithms. Smaller organizations lack access to such resources RLHF5 . A related concern regards who should have access to powerful RLHF models produced by organizations. If RLHF models are open-sourced, it may be difficult to check harmful applications and enforce regulation RLHF5 . Yet restricting access via closed-source models could exclude access to select groups, reducing equity. Likewise concerning is the use of RLHF for weapons development—e.g., better missile systems and more lethal drones. This is a concern for most AI technologies, and global regulatory action must be taken to mitigate possible harm. Lastly, it must be noted that RLHF methods are still susceptible to generic ML vulnerabilities such as adversarial attack, which may affect its ability to enhance industry applications CSET3 . Awareness of all of these possibilities is critical as RLHF continues to develop.

RLHF will impact the degree to which different jobs are susceptible to automation. Although many uses of RLHF are still nascent, these developing applications can provide insight into key implications of RLHF for the workforce. With better models that can be more efficiently used, RLHF advances the narrative that RL-based technologies will quickly close the gap between automation and the dexterity and mobility required for low-wage jobs ME2 . This holds especially for domains in which robotic manipulation and navigation are becoming more dominant. Even so, RLHF is unlikely to lead to full automation of jobs ME2 . Importantly, RLHF methods can automate tedious or high-risk portions of manual labor ME2 , especially for tasks which are dangerous or difficult for humans to complete even if they have the correct intuitions RLHF2 . Humans may guide AI systems in such contexts by providing feedback on how to best complete such tasks RLHF2 . This can enhance workforce safety and morale and does not fully remove humans from the equation, instead shifting human expertise to different areas of production.

RLHF may further affect the spatial distribution of jobs in the workforce. Resulting automation may move jobs, dependent on factors such as required expertise and closeness to service providers ME7 . Job relocation in such contexts is not necessarily constrained by national boundaries, exemplified by techniques involving offshoring of automated operations, which, while cost effective, may introduce regulatory challenges, reduce domestic jobs, and impact transparency ME7 . Future regulations on AI technologies will likely impact the extent to which such impacts are realized.