More than Marketing? On the Information Value of AI Benchmarks for Practitioners

Evaluating AI systems is necessary for understanding their capabilities, limitations, and potential for positive and negative impacts (saxon2024benchmarks). However, this process is complex, requiring careful consideration of which evaluation methods are used and how their results are interpreted.

One of the primary challenges in AI evaluation is ensuring the closeness of an evaluation to a system’s anticipated use (raji2021benchmark; beede2020human). This issue is exemplified by a study conducted by (lebovitz2021ground), which examined five different AI tools deployed in a U.S. hospital. The study found that none of these tools met expectations created by accuracy measured using expert-labeled ground truth data. This highlights the risk of treating constructed quality measures as objective markers of knowledge and underscores the importance of diligent, use-case specific expert evaluation (lebovitz2021ground). This phenomenon of metrics used to evaluate AI models lacking correlation with users’ subjective experiences has been documented across applications. gordon2021disagree note that models which score highly on standardized tests have been observed and criticized for their obvious mistakes. While bridging the gap between assessment and reality is an inherent issue with AI evaluation (kadian2020sim2real), we note that it is possible to mitigate the issue to some extent through careful, case-specific design (grabb2024risks; reuel2024betterbench).

Another challenge in AI evaluation is the inherent difficulty of clearly defining the capabilities AI systems are supposed to exhibit and, subsequently, ensuring construct validity (cronbach1955construct) with respect to the test tasks. While performance on certain tasks like image classification or speech recognition can be clearly-defined, more complex tasks - such as decision-making under uncertainty or ethical reasoning - are much harder to pin down (bostrom2020public). Since AI systems often operate in environments that are unpredictable and involve nuanced human behaviors, it is difficult to set clear, measurable objectives for evaluation (floridi2024three). In a study of the challenges faced by software engineers developing AI systems, patel2008examining find that evaluating performance is a significant challenge, in part due to the trade offs between different attributes, e.g., accuracy and the protection of privacy. That the former is more easily captured than the latter does not make it a better optimization objective.

Further complicating the issue of assessment, AI systems tend to excel in narrow, known tasks, while struggling to generalize to new, unseen scenarios. Evaluations struggle to capture this property, as they are typically based on datasets that represent a small slice of reality. As a result, evaluations can suffer from overfitting, wherein systems perform well on dataset that resemble their training data but fail to generalize when tested on others. This challenge exposes the limitations of controlled, static testing environments that cannot capture the full variability of real-world situations (gebru2021datasheets). The failure of AI systems to generalize well in diverse and dynamic settings underscores the difficulty in designing evaluations that meaningfully assess robustness and adaptability (bender2021dangers).

The difficulty of assessing AI models is compounded by the fact that some systems are not static; they continuously interact with dynamic environments and evolve based on new data and interactions. Evaluations conducted at a single point in time may fail to capture how these systems will perform under changing conditions or how they will adapt to new challenges (brahmakshatriya2023d2x). This makes it difficult to design evaluations that account for the evolving nature of AI, as static tests cannot anticipate future states or emergent behaviors in the system (boyarskaya2020overcoming). Furthermore, many AI systems operate in highly contextual environments where their performance can depend heavily on factors like user behavior, social norms, or regulatory provisions. In their work on evaluating interactive AI systems, boukhelifa2018evaluating find that evaluating such models poses unique challenges: a system may respond to user preferences, but the users themselves alter their behavior based on system performance, leading to co-evolution effects that are difficult to untangle.

To understand how benchmarks became a popular approach to AI system evaluation, we examine their history. Benchmarks originated as measures of computer hardware performance, intended to facilitate selection between an increasingly varied field of products (lewis1985hist). lewis1985hist asserts that there is an important difference between benchmarking in the public and private sectors: the former is required to fairly and thoroughly compare potential vendors, thus placing a high importance on standardized benchmark assessments, while the latter prioritizes profitability and may favor cheaper and less rigorous approaches. As chronicled by (church2018trends; liberman2010jelinek; raji2021benchmark), both the public and private sectors drove the development of benchmarking for Natural Language Processing (NLP) tasks in the 1980s: DARPA funded the development of a well-defined, efficient evaluation for speech-to-text systems (church2018trends). This had a significant impact on the development of the field. As church2018trends writes, a key benefit of this system was that it enabled hill climbing – no longer limited to infrequent evaluations, researchers could measure and pursue more rapid advancements. IBM similarly worked to develop the Common Task Framework (CTF), which would become a precursor to modern AI benchmarking (liberman2010jelinek). By providing well-defined goals and corresponding tasks, the efforts of IBM and DARPA lowered barriers to entry, driving growth of a field (liberman2010jelinek).

We draw further insights about benchmark use from a review of benchmarking literature from other fields.

(cheng2022transistors) highlight the need for standardized benchmarks in the evaluation of field-effect transistors (FETs) to enable consistent and fair comparisons across different studies, given the complexity and variability of device structures and materials. According to the authors, the purpose of a benchmark is to provide a reliable framework for assessing key performance metrics, such as current, mobility, and contact resistance, to help guide the development and optimization of FET technologies. They note that benchmarks can also aid in decision-making by offering a clear basis for identifying the most promising technologies and guiding research efforts toward scalable and high-performance solutions.

(chapman2018environmental) discusses the purpose and limitations of environmental quality benchmarks (EQBs). The primary function of EQBs is to serve as a tool to assess potential harm from chemicals and other stressors in aquatic ecosystems, though the author emphasizes they are not perfect and should not be used alone for final decision-making but that ”they can provide understandable scientific input to decision makers”. According to the author, the benchmarks are designed to filter out cases of negligible concern and highlight potential risks, aiding decision-makers in identifying priorities and guiding further investigations.

Finally, (aniba2010bioinformatics) discuss the role and purpose of benchmarks in bioinformatics, particularly focusing on their application in multiple sequence alignment. The primary function of a benchmark, as the authors state, is to provide an objective, comparative evaluation of computational tools by testing them against standardized datasets. This allows for the identification of strong and weak points, measuring improvements, and helping non-specialists choose appropriate tools. The authors also highlight that benchmarks in their field aid decision-making processes by enabling developers to refine their tools and helping users identify the best method for their needs, as benchmarks represent ”a tool that is used to measure the ability of a software system to meet the requirements of the user”.

Theories of information technology (IT) adoption study how new IT – which broadly encompasses computers, software, data and information processing, and artificial intelligence – becomes accepted and used. AI benchmarks themselves commonly combine software and datasets; as such, we consider them to be a part of IT and draw upon these theories to provide a framework for understanding when and how AI benchmarks are used.

davis1986tam introduces the Technology Acceptance Model (TAM), which identifies two primary factors that impact the acceptance of information technology: perceived usefulness and perceived ease of use. The former refers to the perception of a given tool’s ability to enhance an individual’s job performance, while the latter refers to the perceived effort that the use of the tool requires. Integrating davis1986tam’s model with several others, venkatesh2003unified presents the Unified Theory of Acceptance and Use of Technology (UTAUT). According to this framework, four constructs determine acceptance and usage: performance expectancy, effort expectancy, social influence, and facilitating conditions. Performance and effort correspond to davis1986tam’s usefulness and ease of use, respectively. venkatesh2003unified defines social influence as an individual’s perception that important others believe they should adopt the technology in question. Finally, facilitating conditions are the degree to which an individual believes that organizational and technical infrastructure supports the technology’s adoption.

Applying this theory to AI benchmarks, we predict that venkatesh2003unified’s four criteria must be met in order for benchmarks to be adopted. Since benchmarks aim to provide information about a model, we expect practitioners must determine this information to be useful in order for the criteria of performance expectancy to be met. The widespread use of benchmark scores in publications and marketing materials (tascilar2023quest; bender2021dangers) suggests that social influence is fulfilled. However performance expectancy, effort expectancy, and facilitating conditions remain open questions that we explore in our interview study. We note that even when all factors are present, the UTAUT model does not provide a prediction about absolute levels of usage. Instead, it offers an interpretation for why benchmarks are or are not employed by practitioners.

This study utilizes a grounded theory approach to investigate the development, use, and impact of AI benchmarks (anderson2005empirical). Grounded theory is employed due to its suitability for exploratory research, enabling the derivation of theory based on empirical data (khan2014qualitative). This method is particularly appropriate for examining dynamic and multifaceted subjects such as AI benchmarks, where established theoretical frameworks are limited or underdeveloped (heath2004developing). A semi-structured interview study was chosen for data collection. Interviews are a direct way to obtain evidence about attitudes, feelings, memories, perceptions, anticipations, goals, values, and the like, making them ideal for exploring participants’ nuanced views on benchmarks (mcgrath1995methodology). Semi-structured interviews provide an opportunity to gather rich, qualitative insights while maintaining the ability to adapt to the unique perspectives of each participant. This method provides flexibility for probing deeper into themes that emerged during the interviews, but were not necessarily anticipated by researchers (adams2015conducting).

In-depth interviews were conducted with participants representing four categories of stakeholders: policy analysts, researchers in academia, researchers in industry, and benchmark users in product development and management. Interviews took place over the course of a month, September of 2024, via the online platform Zoom. The interviews were conducted in English and all interviewees were located in the United States.

Participant selection for this study followed a combination of intentional sampling and snowball recruitment. Initially, individuals were identified based on their recognized expertise and relevance to the subject matter, ensuring the inclusion of participants with diverse professional backgrounds. In addition, snowball sampling was employed, wherein participants were asked to recommend other potential informants who could offer insights. This was complemented by posting invitations on LinkedIn and X, which allowed reaching a broader audience of professionals. Finally, the recruitment process involved directly contacting experts through professional networks. Our work intentionally varies participants’ backgrounds to allow for a broad perspective on the development, application, and impact of AI benchmarks across different domains (see Table 1). We note that our sampling approaches resulted in a population that was heavily skewed in terms of gender, thus limiting the generalizability of our conclusions.

All participants provided informed consent, and their identities were anonymized in the final dataset to protect their privacy. IRB approval was obtained prior to data collection, ensuring the study adhered to relevant ethical guidelines for research with human subjects (IRB-76636).

The interview design for this study employed a semi-structured format, allowing for both consistency in data collection and flexibility for exploring emerging themes in greater depth. Interview questions were crafted to align with the study’s objectives of examining the development, application, and impact of AI benchmarks across different domains. This approach enabled the participants to share their experiences while allowing the interviewer to follow up on specific insights as they came up. The interview guide (see Section A.2) was structured into multiple key sections, with questions designed to gather comprehensive data on participants’ interactions with AI benchmarks and their perspectives on their effectiveness, limitations, and improvement suggestions.

We began our interviews with introductory questions such as, ”Please describe your current role” and ”How do you interact with AI models in this role?” These questions were included to establish a baseline understanding of the participant’s professional background and their direct engagement with AI systems. By gaining insight into each participant’s unique context, the interviewer could better interpret the interviewee’s responses to subsequent questions about benchmarks.

Next, the interviews moved to questions like ”In the context of your work, what is the purpose of AI benchmarks?” and ”How satisfied are you with the ability of existing benchmarks to fulfill this purpose?” which were designed to probe participants’ views on the functionality and relevance of benchmarks in their professional tasks. These questions were critical for assessing the alignment between current benchmarks and their intended goals. Additionally, questions such as ”Can you describe your process for finding a suitable benchmark?” and ”What factors are you considering when choosing a benchmark?” aimed to capture the decision-making process involved in benchmark selection, highlighting the criteria and challenges faced by practitioners.

Further questions, such as ”What risks and benefits do you associate with using benchmarks for decision-making?” were designed to understand the broader implications of benchmarking practices, including their impact on outcomes and the potential risks associated with relying on benchmarks. The inclusion of these questions provided a balanced view of the perceived benefits and limitations of benchmarks, offering insights into potential areas for improvements. The structure of the interview guide allowed for a comprehensive exploration of AI benchmarking, while maintaining flexibility to explore unanticipated but relevant themes that arose during the conversations.

Before formal data collection, the interview protocol underwent pilot testing with three participants. Post-interview team debriefs identified which questions worked well, which were unclear, and which elicited unexpected but valuable themes. To achieve this, the team engaged in a structured feedback analysis process (mcgrath1995methodology). Interview questions were assessed for clarity, relevance, participant engagement, and depth of response. Team debriefs and revisions were logged for transparency. Based on this feedback, we refined the interview guide, rewording and reordering questions to maximize clarity and relevance. All interviews were recorded via Zoom and transcribed using Otter.ai.

Data was analyzed using grounded theory coding. Interviews were scheduled for 45 to 60 minutes and transcribed, resulting in a total of 517 pages and 141,702 words, with the per-interview word count ranging from 3,932 to 10,926. During the analysis, we extracted 521 passages (primarily individual sentences) and the process began with open coding (corbin2015basics; muller2014curiosity), where key themes and concepts were identified from the raw interview transcripts.

As the analysis progressed, axial coding was employed to explore the relationships between these themes, leading to more cohesive categories. The final set of 13 high-level code categories and 85 codes including descriptions and the number of corresponding tags in the interviews can be found in Appendix A.1. The analysis followed an iterative process, with the team continually revisiting and revising the codes based on emerging data. A thematic analysis approach ensured that the study captured the richness of the interview data while building a grounded theory that reflects the participants’ experiences. Multiple coders were involved in the process to ensure rigor and reliability, with coding discrepancies resolved through regular team discussions. We used Otter.ai for transcription and Taguette for qualitative coding and thematic analysis. These tools facilitated efficient handling of the data while allowing for transparent tracking of revisions and changes during the coding process.

Across all three settings considered – research, product, and policy – participants reported that benchmarks were highly useful as standardized measures of relative progress towards a benchmark’s target metric.

I-11 describes the ability of benchmarks to evaluate whether a new method is an improvement over previous ones:

”So to make progress, when you’re designing a method . . . you need to have a sense of if you’re actually heading in the right direction. And I think benchmarks are often nice if they actually do encapsulate the problem you care about.”

The relative assessments benchmarks provide can guide future development by indicating the efficacy of current approaches. I-15 describes benchmarks as a directional measure: ”We’re using them as sort of a vector, right? Are we headed in the right direction?” This type of insight is also beneficial in a product setting. I-6 lists questions benchmarks can answer about a new model:

”How do we do [with a] previous version of a model compared to a new version, or . . . if you’re doing a feature improvement, how does the addition of a feature improve or decrease the performance?”

Practically, these signals guided model developers and users in decisions such as whether to adapt the model (e.g., if benchmark scores dropped precipitously) or to continue development (e.g., if it outperformed some baseline). These relative scores were more often used as negative indicators than positive ones. In other words, poor benchmark performance was more likely to disqualify a model from further development or use than good benchmark performance was to qualify it. I-8 describes the impact of benchmarks on model development:

””We’ll see regressions [in score] and then that’s led to some debugging. For instance, people say ’why did numbers get worse on this dimension?’ So they go back, ’Oh, actually, we made a mistake’ . . . Or, actually, you know, we had tried a more aggressive approach for this thing . . . and we need to roll things back. . . . Yeah, it’s called some late nights for someone on our team too, where they find out, like, . . . oh, actually, the results weren’t as good as we expected, so we need to try to slip in another checkpoint before things get released.

In some cases, negative performance could permanently halt a model’s development. I-18 called the impact of significantly poor benchmark performance a ”hammer blow,” since it could effectively terminate a project.

Across all three settings, participants agreed that the standardized, relative measures benchmarks provided were useful as an initial indicator of model performance. They also aligned in considering a model to have failed if its performance significantly dropped in comparison to some baseline. However, when benchmark scores improved, research benchmark users differed from those in product and policy in how they answered the question: is better good enough?

The observed discrepancy in benchmark interpretation may arise from the fact that unlike decisions between models made in research settings, those made in policy and product often require absolute signal due to their potential for direct impact on people. Many of the participants we spoke to found publicly available benchmarks lacking and were actively developing benchmarks suited to the specific deployments they were concerned with. Based on our analyses, we identify the gap between what existing public benchmarks assess and how models will be practically used as a key factor that limits their meaningful adoption. I-9, who was building benchmarks tailored to specific industrial applications, cited the fact ”that benchmarks today are not based on the end user behaviors which are most interesting or especially the most valuable” as a ”key deficit.” Assessing the examples within popular public benchmarks, I-7 found them to be not only irrelevant, but also stated

”Some of the major benchmarks that people use today, MMLU, for example, . . . just go through the questions and look at them, and you’ll realize that they are awful. Just like, . . . terrible. Totally, a bunch of totally irrelevant stuff, things that are totally wrong. ”

While some participants developed their own benchmarks to address issues of quality and relevancy, others forewent benchmarks entirely. Although I-5 noted there were no existing ”benchmarks for the kinds of data that . . . end customers care about,” this did not lead his company to develop its own benchmarks. Due to the level of detail required to understand their models’ performance, the company decided not to ”do anything programmatically today, because oftentimes, [to] improve, the changes can be . . . somewhat nuanced.” Manual evaluation of these specific qualities was seen as more useful than automated evaluation of general ones.

Even when satisfied with available automated metrics, participants in all three groups agreed that benchmarks should be complemented by, rather than substituted for, human evaluation. I-7, who found benchmarks to be highly suited to his use case of ASR, said

”That would be the ideal scenario where you get someone speaking or each language, which we cover, and asking them to spend, like, a couple of hours giving them some instructions where they just absolutely go ballistic at the model, do all kinds of things and see that you will get, like, a good trend.”

When assessing the medical models they were developing, I-14 found that ”keeping humans in the loop is necessary.” Human feedback could supplement both benchmarks and approaches such as model-based red-teaming. A robust system, I-8 reported, would ”have . . . red teaming, and also just . . . people [to] test out the models, like, use it in whatever way, as additions to benchmarks.” Participants agreed that there were insights which could only be gained through direct, manual evaluation of the models.

An additional benefit of such human evaluation is that it is durable in the face of rapid model advancement. While benchmarks frequently saturate, human evaluation remains relevant. I-2, whose work concerned the assessment of future model capabilities, said

”I place a high value on, like, the human interaction, vibes-based evaluation technique, because it’s. . . more or less the only thing that doesn’t get invalidated by a new paradigm or type of model. We don’t really know what models will be like in like five years or something, and it would be very surprising if many of our benchmarks were still used then.”

I-11 named human-centered evaluation as the most significant gap in existing benchmarks:

”Benchmarks. . . measuring like, how LLMs or just generally, models interact with humans are missing. It’s hard and there’s not really a clear setup. I think people probably need to partner more with . . . psychologists and social scientists on how to like do these benchmarks. But those would be the most valuable.”

Participants agreed that task-specific evaluations, designed with a model’s use case in mind, were more helpful than generic ones. Consistent with this belief, they expressed that for models directly used by humans, human evaluation is irreplaceable. Although some participants validated their benchmark results with extensive human evaluation, such testing was most commonly performed by the developers themselves, who interacted with their own models during the development process. Many noted the expense of human annotation as a significant deterrent for collecting comprehensive manual evaluations.

Our work seeks to understand how and when AI benchmarks are used by practitioners. Deciding to use a benchmark can be considered the acceptance of a technology; thus, we return to the UTAUT framework and discuss which of its criteria were or were not met for our participants. Applying this framework, we see that although public benchmarks were generally considered easy to use (effort expectancy), were widely accepted (social influence), and generally came with infrastructure to support their use (facilitating conditions), individuals working in policy and product did not believe them to provide sufficient information to be helpful (performance expectancy). In the following section, we examine factors which contribute to this low performance expectancy and propose changes to the benchmark design and development process that can increase relevancy and improve performance as real-world measures, thereby enhancing both adoption and quality.

More general models have been matched by broader benchmarks testing general, increasingly abstract model capabilities; however, such evaluations face significant issues of construct validity and real-world applicability (raji2021benchmark; cronbach1955construct). Assuming that construct validity in general evaluations was no longer a problem, the question remains: are general benchmarks the most useful approach? When considering a candidate for a specific job, their experience with and approaches to its intended tasks are more important than broad traits such as intelligence. Similarly, if models are to be deployed for particular applications, then it is these use cases which most require evaluation, rather than more general capabilities. As discussed in Section 4.2, our participants desired and often worked to create benchmarks that were more task-specific or real-world use-case inspired, since they found broader benchmarks did not meet their needs.

Section 2.2 describes how benchmarks as we know them today originated from a need to make standardized comparisons between vendors considered for government contracts (lewis1985hist; raji2021benchmark). Although hardware contexts still present challenges of ecological validity (lewis1985hist), evaluation of AI systems poses even greater difficulties. Hence, in the absence of application-oriented designs, benchmarks can only provide relative signal. Across research, product, and policy, this was our participants’ most frequent use case for AI benchmarks. They were confident that benchmarks could demonstrate improvement, but almost always required information and assessment beyond benchmarks when assessing absolute capabilities or properties to make substantive decisions. This did not make benchmarks useless – as I-15, an academmic researcher said, ”It would be impossible to do my job . . . without them.” He explained this necessity in terms of standardization: with benchmarks, ”the whole community knows you measure the same thing.” Furthermore, many participants identified benchmarks as a key factor in driving AI advancements. ”The field [of AI],” I-15 stated, ”has progressed because our ability to make benchmarks.” The mere existence of a benchmark could cause a subfield to advance, since, as I-14 stated, ”Having a public data set . . . attracts technical people to work on that problem.”

Despite their power to drive fields forward, participants remained hesitant to use them for substantive decisions and identified overreliance on benchmark scores for decision-making as a substantial risk. I-9 outlined this danger:

”The . . . most obvious risk is that there’s a disconnect between [a] domain level benchmark and a task specific benchmark. What you see at a very general level is a trend that doesn’t apply to your actual use case. Another issue that we see a lot of is kind of like understand[ing] interpreting the results effectively. So you have a benchmark of legal performance, and the models are all performing in some range of like, 70 to 80% does that mean that they are doing very well? We should rely on them, or should we wait until they get closer to 95, 99% before they’re acceptable for use. Additionally, like, how does it compare to human performance? If humans are doing like 60% then . . . they’re actually doing very well”

Under the UTAUT, this quote reflects the principle of performance expectancy. Participants were only comfortable adopting benchmarks under narrow circumstances, reflecting their doubts about the insights these assessments could provide.

While benchmarks lacked performance expectancy, their acceptance was heavily supported by social influence. venkatesh2003unified defines this factor as ”the degree to which an individual perceives that important others believe she should use the new system.” Benchmark use as an AI community norm is reflected in both the necessity of using benchmarks for research publication and the reporting of these scores in model marketing materials openai2023gpt4; anthropic2024claude3. I-13 noted that ”if you’re evaluating LM capabilities, you’re going to go use like MMLU and MT bench and AGI Eval, because everyone uses them” but not because they hold inherent value. Social influence may compel model developers to run benchmarks, but it does not necessarily drive model users to rely on these scores. I-19, a venture capitalist investing in AI companies, noted that in addition to being a poor signal for the success of an early stage company, benchmarks were unhelpful since ”public benchmarking has become like propaganda.” The fact that participants generally did not use benchmarks to make their ultimate decisions and highlighted the need for additional information contrasts with the way benchmark scores are often used in marketing materials. This finding is consistent with the UTAUT model – according to venkatesh2003unified, performance expectancy is the most important criteria for technology adoption, while social influence has more complex and less consistent effects. Benchmarks may have been accepted as a preliminary measure; however, nearly all participants in product and policy supplemented them and some dismissed them entirely.

Despite benchmarks’ widespread use and well-known limitations, participants who developed benchmarks in both research and product settings found that the impact and difficulty of their work was underestimated. I-8 described his colleagues’ perception of benchmarking: ”I don’t think it’s appreciated enough. I think people say, like, just come up with the data set. How hard can it be?” In research, I-15 expressed that such attitudes led some students away from benchmark development:

”My biggest gripe . . . is going to be that our field only exists and progresses because of benchmarks. Like, if you don’t publish experimental results on a benchmark, you don’t publish like, you literally don’t publish. You will get rejected. Yet, if you try to publish a benchmark data set at a conference, it’ll more likely than not be rejected because you didn’t present a method . . . So if you . . . propose that let’s build a new benchmark, everyone will be like, well, let’s hold on a second. Maybe that’s not worth the time . . . There’s no ROI on that for a PhD student.”

Understanding the difficulty of developing benchmarks is critical, since the problems with existing methods cannot be solved without the investment of substantial time and resources. Practices like the inclusion of community members and domain experts in the benchmark design process are costly, but are essential to evaluating model performance in a way that is truly meaningful to those that models most directly impact.

Many of the participants in our study were working to mitigate the outlined issues with existing benchmarks by developing new, internal ones. When asked about how they addressed these challenges, participants emphasized that a good benchmark should offer a meaningful evaluation of an AI system’s capabilities, aligning closely with real-world applications to ensure robust, meaningful assessments. I-2 mentioned that benchmarks ”should be actually informative about [a] particular capability” and I-7 stressed the importance of ”making sure the data is realistic.” I-7 emphasized that ”there’s a ton of little domain-specific things that matter a lot to getting a good benchmark,” highlighting the need for benchmarks to address the specific contexts in which AI models will be deployed, and the necessity of involving domain experts in the design process.

Another trait that participants emphasized was that of transparency in benchmark scope and goals as a step towards preventing misinterpretation. One way to make scores easier to interpret, which participants frequently mentioned, was providing a human baseline for comparison. For instance, I-2 highlighted that ”assessing the model’s capability relative to the amount of time it would take a human to complete a similar task” was useful. Without this context, the interpretation of a benchmark result was seen not only as difficult, but as a potential risk when unclear results were used for downstream decisions.

The consensus was that benchmarks need to be carefully designed to match the distribution of real-world data and produce statistically significant, reproducible results. Diversity, data quality, and an appropriate level of difficulty were viewed as important factors in ensuring that benchmarks remain informative and adaptable to evolving AI capabilities, avoiding the trap of saturating too quickly:

”Having a benchmark that you can get signal on today, where models capabilities vary in a meaningful way, but that won’t saturate very soon, either is, I think, quite useful, and we’ve seen with a lot of benchmarks, that they saturate quite quickly. So this just means having a wide range of difficulties for your tasks included in the benchmark.” (I-1)

Moreover, participants acknowledged several ongoing challenges in the design of effective benchmarks. One key issue is the difficulty in defining clear, interpretable metrics, and I-8 noted about their internal evaluations that

”We also rarely just report one number, because a lot of this is trade-offs, like one number goes up while the other number goes down, for instance. So we shy away from that. It drives a lot of the developers nuts, right? They just want a single number to optimize, but then there’s Goodhart’s law. […] Okay, what’s the final number? Like, there’s no final number.”

Another frequently voiced concern was the contamination of datasets and the need for proprietary data collection methods to mitigate this risk, which was mentioned by participants as a key reason for not releasing internally developed benchmarks:

”[We are] actually trying to make sure that when we collect data sets, we keep them proprietary, we don’t release them publicly, and we make sure that all of the model providers we use will not retain or retrain on any of the data.” (I-9)

These findings are in line with previous work on AI benchmark quality: (reuel2024betterbench) have shown that many public benchmarks do not take preemptive measures against contamination, outline how benchmark results should be interpreted, or thoroughly explain how knowing about the tested concept is helpful in the real world. Similarly, (liu2024ecbd) find that existing benchmarks often lack validity, which impacts their real-world usefulness.

One overarching challenge respondents emphasized was that the science behind robust evaluations and benchmarks is ”an open problem. How to figure out, like figuring out what’s a reasonable benchmark?” (I-8). Our findings suggest that deliberate design processes, including the improvement points stressed by participants, are necessary to ensure that benchmarks provide meaningful evaluations to practitioners.

Ultimately, benchmarks must move beyond being a ”marketing strategy” to serve as robust tools to practitioners for evaluating AI systems in a transparent, informative, and interpretable manner that reflect meaningful improvements in model capabilities rather than artificial performance gains.