What I Learned Building a Psychometric Test about War Speeches

Do we know what our speeches do to the brains of our audience members? Do our words trigger specific logical processes that govern reason, or evoke images that generate certain emotions, or illicit memories that impact different structures of their personality?

No, of course not, we have barely a clue. There are still A LOT of things we don’t know about the human brain, and as a result, rhetoric and persuasion remain highly mysterious arts. The very questions themselves sound a little science fiction-y. But maybe it doesn’t have to stay that way. Maybe speechwriters can work with psychologists, including cognitive scientists, to explore the workings of our art form. And these insights could have some profound effects on our profession, making us much better at what we do, and more responsible when we do it.

One easy way to start this process is to encourage speechwriters to master psychometrics.

Case-in-Point: Developing a Psychometric Test about War Speeches

My experience started in Fall 2021, as I was taking courses in the Psychology Masters Program at the Extension School at Harvard University (an excellent low-residency program, you should all research it). After writing speeches professionally for 14-years, I decided to transition to academia, with my ultimate goal being entrance into a PhD program in cognitive psychology so I can study psychological warfare.

One of the courses I took that semester was in psychometrics, which is the field devoted to measuring mental capacities, characteristics, and processes – primarily by the development of assessment tools like tests. There are thousands of psychometric tests out there, measuring traits such as intelligence and mental aptitude, and gauging personality. You’ve probably encountered such tests dozens of times in school and throughout your career.

For my final project, I constructed a test that I hoped would measure the confidence a person had in saying “No!” to high demand requests. In the case of this project, I chose rousing war rhetoric and speeches. I chose such speeches because they are, arguably, amongst the most high-demand public requests that can be made of a populace: take part in this activity that will use many societal resources and lead to deaths, possibly your own.

The construct that the test measured – every psychometric test measures a “construct” or designated mental trait, function or process – I dubbed “willpower confidence.” I believed that this “willpower confidence” construct I theorized was important because it would give us insights into how mentally resilient people were in the face of aggressive demands. Our society is filled with aggressive demands with people pressuring us to buy, to vote, to watch, to like whatever it is that they want us to buy, vote, watch and like. Having the ability to say “No!” in the face of such demands is an essential element of Democracy and a necessary ability for any individual to affirm their personal agency.

I then developed a test to measure this construct, which I named the Fast/Slow War Rhetoric Response Test and administered amongst my social media networks in November using the Alchemer survey platform. In this test, participants read 16 speeches from Western History, from such figures as Queen Elizabeth I of England, Prime Minister Winston Churchill and General George S. Patton. They were broken down into two sets of eight speeches each.

For one set, participants were asked to read the speeches quickly and to rate on a scale of “0” to “5” how much effort it would take to say “No.” A rating of “0” meant no effort and “5” signified “extreme” effort. This set measured the impact of persuasion on our “fast” thinking processes, also known as “heuristic” thinking – many pieces of popular rhetoric target this part of our thinking because it often involves short-cuts like stereotypes and emotions.

The second set asked participants to read the speeches more slowly and deliberately, and then to rate, again on a scale of “0” to “5,” the number of counterarguments they believed they could make in response to the speech, with “0” signifying “No counterarguments,” and “5” signifying “At least five counterarguments.” This set was designed to measure the impact of persuasion on “slow” “deliberative” thinking, which is believed to be more precise and logical. (To learn more about the differences between “slow” and “fast” thinking, I recommend the works of Daniel Kahneman.)

Long story short, thanks to the help of many of you on my social media networks, I learned that the “willpower confidence” construct I theorized might possibly exist and that it may share elements with the psychological process of “coping,” the way we deal emotionally with life-challenges. I thank all of you for that.

Of course, the test was a prototype and far from perfect. First, it was too damned long. Asking people to read 16 speeches was simply too much, with some people quitting the test early on. Also, more work was needed to organize the different speeches in terms of rhetorical style. There were many, many other issues, of course. It was a prototype.

I’ve attached a link below to the paper I wrote for the Harvard psychometrics course (which I submitted on December 15th, 2021 and for which I got an A). The paper has a full breakdown of the design of the test, the speeches used, the results, as well as the problems I believed needed addressing for future versions.

I was ready and excited to post the final findings of the test, my experiences as well as my suggestions for you to develop your own such tests—and then Russian forces invaded the Ukraine in February.

WAR, THE SCIENCE OF PERSUASION, AND OUR JOB AS ENABLERS OF POWER

I’ll be honest about my response to the Russian Invasion of the Ukraine with regards to posting my findings and experiences with the Fast/Slow War Rhetoric Test: I chickened out and did nothing.

War in real life is devastating, destroying lives, shattering families, communities and societies and creating heartbreak and atrocity that will forever haunt millions. The social media environment resulting from a war can also be chaotic and conflicted, as well as rife with emotion and language-based combat. I held back from discussing my tiny test and paper in this social media environment because, firstly, I was scared of spurring highly emotional flame wars, and second, I didn’t want to do anything that would distract from the more important issues – such as reasoned debate over what we can and should do as individuals. I still think such reasoned debate remains important on this subject.

As I sat on the findings of my little test, I thought about speechwriters using tools like psychometric tests to pinpoint the building blocks of their persuasion techniques. I decided that the idea could be significant for our profession. The learning curve wouldn't be that great. If you have experience constructing polls then you can learn to craft basic psychometric tests.

Initially, I just imagined developing psychometric tests as a tool for better understanding your own speechwriting and persuasion strategies. Each of us write different kinds of speeches, that have different kinds of cognitive impacts on our listeners. Better understanding these processes would be invaluable in terms of understanding what we do well, and what we don’t. I still believe in the value of just such an application, but I think we can be more ambitious.

At the risk of sounding pompous, we speechwriters need to better understand our role as enablers of those who would be powerful, of the people who ultimately want to make big demands of the public, even declarations of war. It may not be a bad idea for some of us to start working with researchers to clear away some of the mysteries regarding what we do. Maybe we can partner with researchers in their studies. Maybe we can build our own public databases, administered by our various professional speechwriting associations, for white papers or other observations devoted to the psychology of our craft. Maybe we can learn how to build psychometric studies to measure the constructs we think are important in the speechwriting process.

Will we clear up all the mysteries of our art form? Who knows? However, the more precisely we understand what we do, the more precisely we will understand the power we are giving to others. This will make us not only much better persuaders, but much more responsible ones as well.

The link to my Harvard Extension School paper on the War Rhetoric Response Test is above. It's posted on the website for my business, Enchanted Loom Productions (I'm not looking for any more clients--at all.). The paper is copyrighted. If you want to use any part of it, please use the following citation: "Fernandez, Thomas. (2021) The Fast/Slow War Rhetoric Response Test: Design, Pilot Test and Analysis. Harvard University Extension School: Cambridge, Mass."