by Amnon Cavari, Akos Mate, & Miklós Sebők
In a representative democracy like the United States, we expect that the policy priorities expressed by politicians on the campaign trail or in stump speeches reflect the same priorities that they pursue while in office. We further expect that politicians would continue their policy commitment in their programmatic messages as well as in their daily activities and speeches. When they do that, we say that they stay on the democratic script.
We test this proposition, focusing on the relationship between the programmatic addresses of US presidents and the daily speeches by comparing the annual State of the Union address (SOU) with subsequent day-to-day speeches, which we refer to as occasional remarks (ORs). Occasional remarks are crucial because they allow the president to show the electorate that they are following through on their promises. They can also serve as testing grounds for new ideas or messages.
Using the American Presidency Project, we gathered all State of the Union addresses and occasional remarks for every president from Harry Truman to Donald Trump. We then coded the documents using the codebook of the Comparative Agendas Project, which defines 20 policy categories. Because of the large volume of documents that made up our dataset (16,523 speeches divided into nearly 2 million sentences), we used a large language model to conduct the bulk of our coding, supplemented with some manual coding used to train and refine our language model.
We used the coded data to test three hypotheses:
- H1: The policy agenda of the most important programmatic speech (SOU) and of routine remarks (ORs) each year will be positively correlated.
- H2: The correlation between the policy agenda of the most important programmatic speech (SOU) and that of the routine remarks (ORs) will steadily decline over the course of the year.
- H3a: Major domestic and foreign events decrease the diversity of the presidents’ routine attention (measured in ORs) relative to that presented in strategic communication (based on SOU).
- H3b: The effect of domestic and foreign events on the diversity of routine agenda would be conditioned on the diversity of the annual agenda in the SOU.
Figure 1. Correlations between speech types by policy topic.
As the above figure illustrates, across the 20 coded policy topics, there’s a strong correlation between the topics that are emphasized in State of the Union addresses and those that subsequently appear in occasional remarks, giving credence to Hypothesis 1. Applying regression analysis, we also found that, as time goes on, the policy topics addressed in occasional remarks diverge from those emphasized in the State of the Union, supporting Hypothesis 2.
As for Hypothesis 3, we found a positive correlation between the diversity of policy topics referenced in the State of the Union and those referenced in occasional remarks; but, in contrast to our expectation, we do not find that major events (e.g., foreign conflicts, natural disasters, etc.) have a major impact on shifting the focus of presidential remarks.
Our results show that, generally speaking, U.S. presidents are staying on the democratic script: The policy priorities that they outline in their State of the Union addresses are the same priorities to which they return in subsequent remarks. By comparing State of the Union addresses to occasional remarks, we’ve shown a link between programmatic and occasional communications that may have broader applicability beyond the presidency. We have also demonstrated the value of using large language models for parsing large volumes of policy texts, as our model’s coding displayed a higher accuracy rating than our manual coders, and at a fraction of the time. There are numerous avenues for building upon the insights outlined here, including examining the relationship between speechmaking and public opinion, how different speech types intersect with the policymaking process, and exploring the populations exposed to these speeches and how they respond to the speeches.
You can read the original article in Policy Studies Journal at
Cavari, Amnon, Akos Mate and Miklós Sebők. 2024. “ Staying on the Democratic Script? A Deep Learning Analysis of the Speechmaking of U.S. Presidents.” Policy Studies Journal 52(4): 709–729. https://doi.org/10.1111/psj.12534.
About the Authors
Amnon Cavari is Associate Professor and head of the Institute for Liberty and Responsibility at the Lauder School of Government, Diplomacy and Strategy at Reichman University, Israel. Prof. Cavari’s main research interests are in the interrelationship between actions of elected officials and public opinion in the United States and in Israel. He is the author of The Party Politics of Presidential Rhetoric.
Twitter/X: @ACavari
Akos Mate is a computational social scientist whose research interests are political economics and quantitative methodology. He is a research fellow at the Centre for Social Sciences, Budapest. He also teaches as visiting faculty at the Central European University, Vienna, and served as a consultant for the IMF’s Independent Evaluation Office.
Twitter/X: @aakos_m
Miklós Sebők is a Senior Research Fellow at the HUN-REN Centre for Social Sciences (CSS), Budapest. He serves as the research co-director of the Artificial Intelligence National Lab at CSS, the principal investigator of the V-SHIFT Momentum research project, and the convenor of the COMPTEXT conference. His main research interest lies at the intersection of policy studies and natural language processing.
Twitter/X: @Miklos_Sebok
Policy Studies Journal Blog 