Framing in the Presence of Supporting Data: A Case Study in U.S. Economic News

National news publications have a massive choice-set of topics they could cover on any given day. Though it may seem like the publications are reacting to a defined set of events that recently happened, outside of a few major events, they choose what the news is on any given day. In other words, the news in mainstream publications does not present an objective view of current affairs. Rather, it is influenced by the selection made by media professionals McQuail (1992). Selection is driven by what researchers have called news values Galtung and Ruge (1965); Harcup T and O’Neill D (2001, 2017), which determine whether a story is newsworthy.

While there is variation in the taxonomies that have been proposed to determine newsworthiness, the most recent ones include factors such as: the power of the elite (stories concerning powerful entities), bad and good news (stories with particular emotional overtones), and magnitude (stories that are perceived as significant for a wide audience) Harcup T and O’Neill D (2017). More interestingly, news values vary from publication to publication. A study of the run-up to the 2022 election showed that the New York Times and Washington Post averaged just two overlapping topics each morning in their printed editions out of about six articles Rothschild et al. (2023).

Conditional on selecting a topic, there is additional leeway on how to cover it, as there is a wide-range of facts and opinions that could spin the takeaway of the readers. This phenomenon, referred to as framing, has been widely studied in communication studies. While there is not an agreed upon definition of framing, recent surveys Vallejo et al. (2023) have identified the following prevalent definitions: equivalence framing (presenting the same exact information in different ways) Cacciatore et al. (2015), emphasis framing (highlighting specific aspects of an event to promote a particular interpretation) Entman (2007), and story framing (leveraging established narratives to convey information) Hallahan (1999).

When it comes to the economy, a publication could present a rosy or terrible version of the economy. They can do so without lying by simply picking the right economic indicator with the right perspective. For example, there are months where the value of the non-farm payroll is above average, but below the market expectation: either perspective is technically true. But, with a relatively finite set of indicators, and an established way for economists to judge them, there is still a clear ground truth to compare if any given indicator with any given perspective represents the consensus understanding of the economy at that time.

Understanding the health and wealth of the U.S. economy is critical for investment and growth. Most modern economic indicators have been tracked for decades, codified in the by the mid-20th century. Examples include: jobs numbers (e.g., non-farm payroll), prices (e.g., consumer price index), and macro economic (e.g., gross domestic product). With the exception of market prices (e.g., S&P500), which are a reflection of the underlying assets, the core indicators are tracked by a few government agencies (e.g, the Bureau of Economic Analysis for GDP, spending, income, trade, and the Bureau of Labor Statistics for unemployment, prices, productivity), which conduct regular, large scale polling.

These economic indicators inform the public on the health of the economy. The job numbers are front page news when they are released at 8:30 AM ET on the first Friday of each month. The state of the markets is constantly covered in the news. Indicators such as the consumer price index pop up whenever the media finds them most important, interesting, and engaging. Most people have limited views of the economy outside of their close social circle, so these values are the key to their understanding of the health of the economy as a whole. And, traditionally, the health of the economy has tracked very closely with the approval of incumbent political leaders Hummel and Rothschild (2014).

This paper builds off of an extensive literature that has not just identified the quantity of economic coverage, but framing of that coverage. Most of this work is geared towards matching the sentiment of articles about the economy to both consumer sentiment and economic indicators, tracking both historical value and predicting future ones Hopkins et al. (2017); Ardia et al. (2019); Shapiro et al. (2022); Seki et al. (2022); Bybee (2023); van Binsbergen et al. (2024). While the methods continue to evolve from simple keywords to BERT or LLM-based predictions, they are focused on article-level sentiment, aggregated to states or the U.S. They also study a mix of questions about economic impact that benefit from a long time-series of sentiment, granular in both time and geography, regarding questions as diverse as the impact of bubbles on industries (Bybee, 2023), economic shocks to economic decisions Shapiro et al. (2022), to the relationship between news and perception of the economy Hopkins et al. (2017). Our paper extends this literature by capturing not only the sentiment of economic articles, but what economic indicators and economic indicator-level sentiment is driving it. This allows us to dissect what editorial choices are driving perceived sentiment around the economy.

Scholars have increasingly adopted computational approaches to study framing, allowing studies to scale to large media repositories. In most cases, researchers adopt unsupervised techniques such as topic modeling to identify latent themes DiMaggio et al. (2013); Nguyen et al. (2015); Gilardi et al. (2021). However, topic models are limited in their ability to capture framing nuances. By defining topics as simple distributions over words, they lack the semantic and discursive conceptualization needed to answer how issues and events are being presented Ali and Hassan (2022).

There is also a body of work leveraging supervised learning Johnson et al. (2017); Khanehzar et al. (2019); Kwak et al. (2020); Huguet Cabot et al. (2020); Mendelsohn et al. (2021) and lexicon expansion techniques Field et al. (2018); Roy and Goldwasser (2020) to analyze framing. For this to be feasible, authors must define a concrete taxonomy of which frames are relevant and how they are represented in data. Most of these approaches rely on the taxonomy proposed by  Boydstun et al. (2014). These dimensions correspond to broad themes such as “economic”, and “public opinion”. While these themes have shown to be useful to model a select set of contentious political issues like abortion and immigration, they are too broad to capture how the economy and other data-driven topics are being presented in the media.

The data for this study came from the Internet Archive’s Wayback Machine Archive (1996-2023). We consider six American publishers: The New York Times, The Wall Street Journal, The Washington Post, Fox News, HuffPost and Breitbart. Selected sources represent a broad range of mainstream media outlets that were easily accessible through the archive. Breitbart is included for a future analysis comparing mainstream media with a fringe source. For each publisher, we collected the “front page” from every Wayback Machine entry between Jan. 1st 2015 to Jan. 1st 2023. For each front page, we recorded articles in the top 10 positions and discarded all duplicates.

To identify articles that discuss the economy, we curated a lexicon of economic terms sourced from the Bureau of Labor Statistics and the FRED database operated by the Federal Reserve bank of St. Louis.^†††https://www.bls.gov/data/^‡‡‡https://fred.stlouisfed.org/ These sources provide a fairly comprehensive list of common metrics for economic activity. We consider an article to be relevant to the economy if it contains at least three sentences mentioning an economic term from our lexicon. This resulted in a total of 199,066 articles (Tab. 1). App. A.1 includes additional details about the data collection process and the lexicon of economic terms.

We employed a group of six annotators to label our dataset following the guidelines outlined in App. A.3. Our annotators comprised senior, postdoctoral and pre-doctoral researchers. Their fields of study include: economics, computational social science, and communications. They received training from the authors of the paper and several rounds of calibration were performed before annotation began. Articles were chosen such that there would be some overlap between coders (in order to check quality) and a decent breadth of coverage. We prioritized topic diversity over uniformity of news outlets and thus followed the sampling procedure introduced in  Pacheco et al. (2022b, 2023).

For each article in their batch, annotators selected: (1) what type of economic information was the most prominent (macro, government, industry-specific, business-specific or personal), (2) the framing of the general economic conditions (good, fair, bad), and (3) the framing of the direction that the economy is heading (better, same, worse). These questions were adapted from the Gallup economic index ^§§§https://news.gallup.com/poll/1609/consumer-views-economy.aspx. Then, for every valid numerical value reported in the article, annotators selected: (1) what type of economic data it reports (macro, government, industry-specific, business-specific or personal), (2) which indicator was reported (e.g., jobs, prices. See App. A.3 for the full list of options), and (3) the reported polarity of each quantity (positive, negative, neutral). We report resulting statistics in Table 1, and inter-annotator agreement in Tab.  2. To calculate agreement, we only consider examples that were annotated by two or more coders. In App. A.4 we include statistics for the number of coders for each of the label categories. Lastly, we show the resulting annotation distribution for all frame components in Figs. 2 and  3.

We find that inter-annotator agreement is generally good for quantity-level information (Krippenforff’s $\alpha$ 0.56-0.83). This reflects our intuition that capturing framing at the level of supporting data points may be a better alternative to relying on general topical markers. On the other hand, article-level judgements which aim to capture main “takeaways” are considerably harder for annotators, exhibiting moderate agreement (Krippenforff’s $\alpha$ 0.42-0.48). This is expected and in line with other high-level framing tasks Card et al. (2015); Roy et al. (2021); Mendelsohn et al. (2021). To further characterize the lower agreement values, we include confusion matrices and some examples illustrating ambiguous cases in App. A.5. We find that most of the confusion lies on the following article-level judgements: (1) identifying the main type of the article, particularly deciding between macro-economic and government-specific types when both types of data points are present in the article, (2) characterizing articles that are positive towards the economy, as they tend to be more subtle than the clearly negative ones, and (3) identifying articles that suggest that the economy is doing neither better nor worse, but staying the same, which is an inherently ambiguous category. We have included a similar confusion analysis for quantitative-level judgements in App. A.5.

Statistical relational learning (SRL) methods attempt to model a joint distribution over relational, inter-dependent data Richardson and Domingos (2006); Bach et al. (2017); Pacheco and Goldwasser (2021). These methods have proven particularly effective in tasks where contextualizing information and interdependent decisions can compensate for a low number of annotated examples Deng and Wiebe (2015); Roy et al. (2021); Pacheco et al. (2022a).

To model framing, we take advantage of the relations between the values of the frame components for a given article and implement a relational model using Probabilistic Soft Logic (PSL) Bach et al. (2017). Dependencies in PSL are expressed using weighted logical rules of the form: $w_{r}:P_{1}\wedge...\wedge P_{n-1}\rightarrow P_{n}$, where $w_{r}$ indicates the importance of the rule in the model, and can be learned from data. Predicates $P_{i}$ correspond to decisions and observations. Rules are then compiled into a Hinge-Loss Markov random field and weights are learned using maximum likelihood estimation. We consider the following rules:

Priors. We explicitly model prior probabilities for all article-level (type, economic conditions and economic direction) and quantity-level (type, macro-indicator, and sentiment) components. These priors are derived from supervised classifiers trained on our labeled dataset.

$(r_{1})$ Consistency between the quantity type and the macro-indicator. We enforce that if the quantity of a type is macro, then a macro-indicator must be predicted, and vice-versa: $\mathtt{Reports(a,q)}\wedge\mathtt{Type(q,macro)}\Leftrightarrow\neg\mathtt{MacroIndicator(q,none)}$

$(r_{2})$ Consistency between the article type, economic conditions and direction. We enforce that if the article type is macro, then economic conditions and economic direction must be predicted and vice-versa: $\mathtt{Type(a,macro)}\Leftrightarrow\neg\mathtt{EconConditions(q,irrelevant)}\wedge\neg\mathtt{EconRating(q,irrelevant)}$

$(r_{3})$ Dependency between the polarity of quantities and economic conditions. Our intuition is that if several negative quantities are reported in the article, then the article is likely to frame economic conditions as negative, and vice-versa for positive quantities: $\mathtt{Reports(a,q)}\wedge\mathtt{Polarity(q,p)}\Rightarrow\mathtt{EconConditions(a,p)}$

$(r_{4})$ Dependency between the polarity of quantities and economic direction. Our intuition is that if many negative quantities are reported in the article, then the article is likely to frame the direction of the economy as negative, and vice-versa for positive quantities: $\mathtt{Reports(a,q)}\wedge\mathtt{Polarity(q,p)}\Rightarrow\mathtt{EconDirection(a,p)}$

$(r_{5})$ Dependency between neighboring quantity types. Our intuition is that there are common patterns in the types of consecutive quantities (e.g., sequential dependencies): $\mathtt{Precedes(q1,q2)}\wedge\mathtt{Type(q1,t1)}\Rightarrow\mathtt{Type(q2,t2)}$

Rule $r_{1}$ and $r_{2}$ are modeled as hard constraints, and are always enforced because they are designed to force predictions to follow the frame annotation structure. Rules $r_{3-5}$ are modeled as soft constraints, and their weights are learned through PSL. This way, we allow for predictions that do not conform to the templates to be active.

Pre-trained language models (LMs) are one of the most effective techniques for acquiring knowledge from unlabeled text data Devlin et al. (2019). Domain-adaptive pre-training (DAPT) further enriches LMs with in-domain data Gururangan et al. (2020). In both of these pre-training stages, learning is conducted by randomly masking words in a large dataset and training the the LM to predict these words. To take advantage of these strategies, we use pre-trained RoBERTA and perform DAPT with our large unlabeled dataset (all 199,066 economic articles collected). We use this LM to fine-tune the classifiers that are used as priors in PSL. This strategy has been repeatedly used to combine PSL with strong classifiers, with consistent success  Sridhar et al. (2015); Pacheco and Goldwasser (2021); Roy et al. (2021).