History of the Broader Impacts Criterion: Providing an Introduction, Historical Description, Nature of Broader Impacts, and Guiding Theory

Introduction:

The term, “broader impacts”, was originally coined in 1996 by the National Science Foundation (NSF) to be implemented as a scoring criterion for grant proposals. However, the origins of the “entire” broader impacts concept have practical and philosophical roots in values and thought established Before the Common Era (BCE). From a humanistic perspective, origins of this concept can also be found in other cultures throughout the world. 

In the context of historical Western ideology and modern United States, some of these ideas and values were articulated by individuals such as Aristotle and Jean-Jacques Rousseau. These ideas and values evolved as they were shaped by other philosophers, theorists, and different types of scientists over time. However, rapid evolution, diversification, amplification, and the eventual restructuring of these ideas and principles occurred when more “societally competent and proficient [1]” peoples and civilizations were encountered in the Americas. 

Ancient American civilizations, which spanned the Paleo-Indian, Archaic, and Formative Periods, had achieved broad integration and utilization of the societal benefiting concept before they were visited and colonized by Europeans. The broad integration and utilization of this idea is well documented through many of the written and oral histories of the Peoples of the Sovereign Nations (the Native Americans and Indigenous tribes), https://www.choctawschool.com/home-side-menu/iti-fabvssa/%E2%80%98iyyi-kowa%E2%80%99-a-choctaw-concept-of-service.aspx [2]. The impact and ramifications of Ancient American colonization and the assimilation, use, and propagation of Indigenous ideas and values is also well-documented by many scholars [3].

All together suggests, that the concept broader impacts is built upon had been used and subsequently spread throughout the Americas well before NSF decided to embody part of the “entire” idea in a criterion. A revolutionary and potential inflammatory statement for the status quo.

For example, the “medicine man”, contrary to what has been portrayed ideologically in Western media, was not a babbling mystic. “In some Nations medicine men were also women”. Depending on the Sovereign Nation, medicine people could also be educators, researchers conducting research, artists, providers, humanitarians, engineers, historians, philosophers, administrators, social scientists, chemists, or biologists. These persons were positioned to benefit society in real tangible ways. For the Peoples of the Choctaw Nation, one way this would have been accomplished is through varying social practices of service by serving others and developing relationships.

This act of creating connections was termed “I’yyi kowa” or (broken foot). As a concept and practice, it meant that this service needed to ensure beneficial stability or “social balance” and advancement of their society. In other words, "people in these professions needed to achieve societal benefits and desired societal outcomes". Note: I’yyi kowa can be spelled differently between the varying Mississippi and Oklahoma Choctaw communities but the concept it represents is the same. Some information on I’yyi kowa can be read about starting on page 188 in, “The Ball Game of the Southeast: Stickball and Cultural Resource Management (CRM) in the Anthropocene”, https://shareok.org/handle/11244/50785, written by Scott Ketchum.

More importantly, this idea of societal benefit was a position taken by many Indigenous communities. The idea was called by different names. It was a required value and a cultural expectation. In addition, similar notions or concepts were, and still are, being utilized around the world. “In this regard, one question that has not been asked, becomes germane to the practical, philosophical, and historical roots of broader impacts and how it relates to the NSF broader impacts criterion”.

What are the best practices that could be gleaned from older civilizations and cultures that used the concept of broader impacts or its aspects as an expectation?

These practical, philosophical, and historical roots of what would become NSF broader impacts are some of the keys to understanding and clarifying the practical, theoretical, and philosophical issues surrounding the NSF broader impacts criterion. These practical, philosophical, and historical roots also provide a context for the criterion discussion on Return On Investment (ROI) and the relationship between Science, Technology, and Society. In addition, the roots/origins or the “Origins of Broader Impacts” concept has the potential to provide insight as to why the concept of broader impacts has been implemented as a criterion or requirement internationally. A discussion about the Origins of Broader Impacts and its implications for ROI, Science, Technology, and Society (STS), and why broader impacts is an international phenomenon will be covered at a later time.

 

International Use of the Broader Impacts Conceptual Aspects as a Criterion

Although broader impacts are an international phenomenon, the academic community has only recently become aware that broader impacts, as a concept, for use by agency, government, or organizational criterion has been implemented in numerous countries world-wide (also see next section). This is because when the concept of broader impacts has been used outside of NSF, it has been coined with diverse names, terms, and phrases. It has also been used in different ways. These different names, terms, or phrases, which embody this concept or aspects of this concept, can be categorized as broader impacts-like.

A few broader impacts-like examples include: Valorization, Benefit to Society & Economy and Value of Public Engagement, Knowledge Mobilization, Equity in Development, Capacity Building, Harmonious Development, National Economic & Social Development and Social Influence, Ultimate Outcomes, and phrases/concepts underpinning Responsible Research and Innovation and the Research Excellence Framework. A related discussion is provided in my Debunking the Twenty (20) Major Misconceptions About Broader Impacts, One at a Time Post. These names, terms, phrases, and aspects of the concept along with their location have been provided below in table 1.

 

Prevalence of the Broader Impacts Conceptual Aspects and Criterion

There are 195 recognized official countries in the world - 196 if one includes Taiwan. An investigation conducted to determine and better understand the universality of broader impacts concept use as a criterion, indicate that these types of criteria are employed in most countries internationally.

Data for this investigation was obtained through a general identification and count of broader impacts-like names, terms, and phrases used around the world [4]. This identification and count was started in late 2014. The results of this count indicated there were at least 145 out of 195 countries that used a broader impacts-like name, term, or phrase. At the time of this count, many of these countries, including the United States, had more than one broader-impacts like name, term, phrase.

This meant that 74.52% of the countries throughout the world had and/or used at least part of a broader impacts concept. Thus, it appeared that there were 25.48% of countries that did not use or have their own broader impacts-like word, term, or phrase. This initial count ended in 2015.

To verify the first count of the broader impacts-like terms, words, or phrases completed between 2014 and 2015, another count was initiated in late 2016 [5]. In addition to verifying the first count, the intent of this count was to be more thorough and to identify every single broader impacts-like word, term, or phrase throughout the world. Note: This count is still in progress.

During this second count, it has been found that several entities from the countries who were in the above mentioned 25.48% had also adopted or had been utilizing a broader impacts-like term or phrase. These primarily came from organizations in countries that employed a broader impacts concept. Altogether, current estimates seem to indicate that close to 82% of the countries throughout the world currently have or use some aspect of the broader impacts concept. 

One implication of this, among many, means that some of the philosophical, practical, and theoretical issues surrounding the NSF Broader Impacts Criterion are not new or unique.

This is a topic of high-importance and will be discussed later. However, the reason that all of this has been stated is to allow everyone to understand and develop an appreciation for the following year-by-year account of the major events related to the implementation of NSF’s Broader Impacts Criterion.

History of The Broader Impacts Criterion:

Before 1996 NSF used four criteria to determine if a proposal would be awarded. The four criteria were: (I) Researcher Performance Competence; (II) Intrinsic Merit of Research; (III) Utility or Relevance of the Research and; (IV) Effect of the Research on the Infrastructure of Science and Engineering. These criteria were first adopted in 1981 and used until 1996.

The re-assessment and subsequent replacement of these four criteria resulted from a combination of events that happened at the national level and with NSF. Congress passed the Government Performance and Results Act (GPRA) in 1993. Information on the GPRA can be found here, https://www.congress.gov/bill/103rd-congress/senate-bill/00020. This now meant that NSF’s goals and strategies needed to be linked to the outcomes of the investments it had made throughout the country.

By the mid-1990s NSF had expanded its portfolio to include several things like broad education initiatives. NSF had also adopted a new strategic plan in 1995 (NSF 95-24). NSF 95-24 embraced some new long-term core strategies and goals. One of the aims of these long-term strategies and goals was about promoting knowledge in service or “benefit” to society. This strategic plan was called “NSF in a Changing World”. Meaning that they specifically needed to exemplify, participate, provide proof, acknowledge, and measure the societal benefit of its agency’s mission. “NSF needed to have a broader impact”! All of which was consistent with the trend happening internationally.

Official re-assessment of these four criteria were performed by a combined National Science Board (NSB) and NSF Task Force. The potential implementation of this term as a criterion was first provided in the National Science Board (NSB) and NSF Staff Task Force on Merit Review Discussion Report. A discussion about NSF’s Intellectual Merit criterion was also provided in this summary, NSB and NSF 1996 Task Force Report.

After the task force’s findings, NSB and NSF were convinced that the effective utilization and implementation of the broader impacts criterion would better allow NSF to connect its investments to societal value and benefit. NSF thought that it would allow them to better demonstrate and more clearly align their goals to their strategic plan. In addition, the use of this term, as a criterion, would allow NSF to combine and replace at least two out of the four criteria used from 1981 to 1996. A result that would also allow NSF to streamline its review process. How these two criteria were combined and reframed in the broader impacts criterion and in general are provided in figure 1. Click here for figure 1 and associated PowerPoint slides.

In 1997 NSF officially introduced broader impacts to the nation via notice 121. The two 1981-1996 criteria were replaced by the Broader Impacts Criterion. NSF stated that broader impacts would be used as one of their criteria for determining if a proposal would be awarded. This new term, as indicated in figure 1, was defined as "encompassing the potential to benefit society and contribute to achievement of specific, desired societal outcomes". The term that replaced the other two 1981-1996 criteria was called Intellectual Merit.

In 1998 and 1999 Congress told NSF to work with the National Academy of Public Administration (NAPA) to determine the effects of their new merit review criteria. The 2001 NAPA report discussed the academic and research community’s views and challenges of the broader impacts criterion from 1997-2000. As J. Britt Holbrook wrote in his National Science Foundation: Second Merit Review Criterion editorial, “NAPA stated that NSF needed to address a host of philosophical issues raised by the broader impacts criterion”. J. Britt Holbrook does an excellent job in this short article detailing many of these issues.

After the NAPA report, NSF began to try to address these philosophical issues. This started about/around 2000. NSF continues to make efforts to address many of these challenges. This has been in part due to the apparent confusion about broader impacts and how it was inconsistently applied as a criterion. This sentiment has continued to be expressed by many across the nation. Evidence of this can be read in three short editorials - one in 2006/2007; the second in 2011/2012; and the third in 2017.

Several events that followed the NAPA report began to shape NSF’s current version of the Broader Impacts Criterion. During the mid-2000’s many started to question the efficacy of and issues surrounding NSF broader impacts. This led to the American Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science (COMPETES) Act of 2007. This is commonly known as the American COMPETES Act.

In the years following there were also other major attempts to address NSF’s Broader Impacts Criterion. Major events that have shaped or been influential in the response to the criterion between 2010-2017 are provided below. Note: the specific nuances outlining how each of these events historically developed will not be discussed.

First, Congress approved the American Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science (COMPETES) Reauthorization Act in 2010. This legislation mandated broader impacts and encouraged Institutes of Higher Education (IHEs) to assist investigators in achieving the Broader Impacts criterion. It also required investigators to provide evidence of institutional broader impacts resources.

In the year that followed, the National Science Board (NSB) provided statements supporting the Broader Impacts Criterion. The NSB stated that NSF broader impacts would continue. NSB also stated that IHEs should provide support for investigators because there was still confusion about the criterion.

Between 2011 and 2012, NSF made some procedural changes which provided nine over-arching long-term outcomes as it related to broader impacts. This is discussed further in Jeffery Mervis’s write-up in Science Mag News online. Some refer to this as Broader Impacts 2.0.

These long-term outcomes, referred to as NSF’s 1-9, that are positioned to yield sustainable positive impacts are provided below in table 2. In the table, outcome number four is considered the “Everything Else” clause. This suggests that outcomes 1-9 are not written numerically by importance. It also means that there is not a predetermined finite or prescriptive way to achieve NSF’s 1-9. Note: the entitled section labelled “NSF Broader Impacts Categories” in table 2 has been provided by the author as a way to group NSF’s 1-9 to explain the main different types of long-term outcomes.

In 2013, the National Alliance for Broader Impacts (NABI) was formed. The National Alliance for Broader Impacts (NABI) received funding through a NSF Research Coordination Network (RCN) grant. NABI is comprised of about 200 institutions, companies, colleges, and organizations across the nation.

The goal of NABI is to create a community of practice that fosters the development of sustainable and scalable institutional capacity and engagement in the broader impacts of STEM research activity. NABI holds a national summit every year. In 2018 it will be hosted by Brown University. The theme for the 2018 NABI Summit will be "Creative Communication and Scholarship".

After the establishment of NABI, the number of broader impacts offices, networks, and organizations that provided help with NSF broader impacts increased across the nation. Conservative preliminary estimates seem to indicate that there are at least 50 NSF broader impacts focused units, networks, offices, or organizations either being developed or have developed across the nation since the establishment of NABI. However, this count could be in the hundreds. More about NABI can be found in this article written by Susan Renoe and Oludurotimi Adetunji.

In early 2015, NABI convened a working group to develop a guiding document for the NSF Broader Impacts Criterion. The purpose of this document was to assist program managers, proposal reviewers, and review panels in their evaluation of broader impacts components of NSF proposals. The guiding document was the first nation-wide/community attempt to begin the process of normalizing the way review panels evaluated and rated proposed NSF broader impacts plans. This document was released at the end of 2015 and has been nationally vetted and used across the nation. The name of this document is shown in figure 2.

In 2016, the American Innovation and Competitiveness Act (AICA) was passed by the 114th Congress. The AICA was considered the successor to the America COMPETES Acts of 2007 and 2010. A brief discussion and the implications of AICA was provided by Mitch Ambrose in the American Institute of Physics’ (AIP) Science Policy News. 

Also in 2016, a workshop about NSF broader impacts was organized by Sheldon Jacobson (from University of Illinois), Jerome Hajjar (from Northeastern University), Dawn Tilbury (from University of Michigan), and Andrew Johnson (from Texas A&M University). This workshop was called, “Setting a Broader Impacts Innovation Roadmap”. It took place in Arlington, Virginia and was supported by NSF (CMMI-1629955). At this workshop, researchers and administrators, mostly from the Mechanical, Industrial, and Civil Engineering disciplines, deliberated about how investments made by NSF could be enhanced by broader impacts contributions.

There have been other workshops, meetings, and seminars on broader impacts. However, this appears to be the first time publicly any group of specific disciplines, specifically several engineering disciplines came together to provide answers to questions about the NSF Broader Impacts Criterion and begin to articulate aspects of the concept. The summary of this workshop and their recommendations are provided in their Draft Report. During the workshop five main questions were discussed in the context of NSF. These questions are provided in table 2. This will be discussed more in a future paper.

Lastly, in 2017, approximately a month after their yearly summit, NABI held a two-day meeting to discuss the Future Directions of Broader Impacts. This meeting brought together NSF officers, policy professionals, university administrators, broader impact and extension professionals, and representatives from two- and four-year institutions. This included Historically Black College and Universities (HBCUs), Minority Serving Institutions (MSIs), and Hispanic Serving Institutions (HSIs). During this time, there were discussions on developing needs assessments, strategies for creating consensus, and how to move broader impacts forward professionally and logistically. One topic that began to surface in this meeting centered on the Research and Scholarship of Broader Impacts (SoBI) [6].

Concluding Remarks:

Many of the above activities were done with the goal to provide better understanding, clarity, and address philosophical issues associated with NSF broader impacts. The actions taken have brought us a long way to reaching this goal, specifically within the context of a criterion. Yet after almost 20 years of actions this goal has still not been obtained. The fallacy here, is not understanding that the NSF broader impacts criterion represents only one part of an older, much larger, and more well-established broader impacts concept.

Broader impacts are an international phenomenon and not solely a function of NSF. Throughout history and in other countries, places, and organizations they are called by a different name. Which suggest that there is underlying theory and philosophical underpinnings that have not yet been sufficiently addressed. To adequately address both the confusion and philosophical issues associated with this criterion require addressing the philosophical tenets of the concept. In other words, we must well-characterize the entire concept and not just in the context of an agency or criterion.

 

Remarks Part I: Introduction to the “Nature of Broader Impacts”

To provide better understanding, clarity, and to address the philosophical issues associated with the broader impacts criterion requires specific explanation as to the “Nature of Broader Impacts”. Characterizing the Nature of Broader Impacts would allow us to ascertain the fundamental principles of the broader impacts concept. An understanding of this and its “Nature” would also inform what steps should be taken to successfully do and advance the broader impacts criterion. Note: to successfully accomplish characterizing the Nature of Broader Impacts also requires an in-depth analysis of the concept’s roots/origins. An examination of the Origins of Broader Impacts would be one major sub-field of Research and SoBI.

What is the Nature of Broader Impacts?

In general, the Nature of Broader Impacts refers to key principles and ideas which provide a description, understanding, and the action of broader impacts as a way of knowing. It also applies to the characteristics of broader impacts knowledge. The Nature of Broader Impacts would be another major sub-field of Research and SoBI.

The Nature of Broader Impacts also begins to provide a philosophical model with distinct Cosmological, Ontological, Epistemological, and Axiological approaches. These approaches can be found, situated, or used in many of the major “Broader Impacts Perspectives”. These major Broader Impacts Perspectives are the Historical, Philosophical, Developmental, Ethical, Conceptual/Theoretical, and Practical. Some of these perspectives stated here could be sub-areas of one another or models themselves depending on the context. All the above-mentioned perspectives, could also be classified as sub-fields of Research and SoBI, figure 3. 

The major sub-field of Research and SoBI that currently receives the most attention is the Broader Impacts Perspectives, specifically the Practical. The Broader Impacts Practical Perspective can include several topics, for example: broader impacts as an industry model; or university sustainability through societal benefit; or achieving societal benefit in the context of an agency. This last topic has been discussed by many in the academy and at NSF.

This topic has also been discussed in peer-reviewed journal articles. Conservatively speaking, there have been 34 articles published in peer-reviewed journals relating to NSF broader impacts from 1997 until September, 2017. The first article that appeared in a peer-reviewed journal was written by J. Britt Holbrook in 2005. The year that had the highest number of peer-reviewed publications on NSF broader impacts was in 2009. This was in the Journal of Epistemology, Special Issue.

Many of the peer-reviewed journal articles on NSF broader impacts, would also be primarily representative of the Broader Impacts Practical Perspectives. However, it is apparent that most of these topics and sub-fields of Research and SoBI, including the Broader Impacts Practical Perspective, have not been adequately explored. The Nature of Broader Impacts, Origins of Broader Impacts, and the Broader Impacts Perspectives should be rigorously examined if advancing and clarifying the criterion and concept is important. How these major sub-fields overlap, intersect, and are framed within the context of Research and SoBI are provided in figure 3. Click here for an enlarged picture of figure 3. This also has broader implications.

Remarks Part II: The Guiding Theory

The analysis of and research into the concept of broader impacts and the NSF Broader Impacts Criterion has led to the uncovering and development of a potential universal theory. These perspectives, approaches, models, and national and international societal benefit requirements are guided by a larger organizing thought called Societal Benefit Theory and Practice (SBT&P). SBT&P may be the guiding and unifying theory over all models, theories, engagement topics, and actions to achieve (or why people try to achieve) a societal benefit. This also includes topics like outreach, community engagement, civic engagement, engaged scholarship, community based participatory engaged research, and engaged teaching and learning.

This is also one reason that could explain why these topics and many other things can be broader impacts and/or be used to accomplish a broader impact. SBT&P has implications for teaching, all occupations, and every person in the world. I will begin to discuss several of these topics, the Broader Impacts Perspectives, Origins of Broader Impacts, the Nature of Broader Impacts, and elaborate further on figure 3 in some of my upcoming post, or Electronic Letters (ELs).

 

Footnotes:

[1] Both societal competence and proficiency are important aspects in achieving, sustaining, and advancing societal benefits or broader impacts in flourishing civilizations. Societally competent refers to a set of congruent behaviors, attitudes, and policies that come together in a system, agency or among professionals and enable that system, agency or those professions to work effectively in and maintain societal beneficial situations. Societally proficient refers to the set of values and behaviors in an individual, or the set of policies and practices in an organization, that create the appropriate mindset and approach to effectively responding to the issues caused by a lack of societal benefit. 

[2] This concept was utilized in the context of what these Ancient American civilizations valued which was different from the primary values held by the Europeans who visited the Americas. Ancient American value systems allowed for quick and efficient transfer of the concept. This would be equivalent to the modern day “Ideavirus” coined by Seth Godin.

[3] There are several references that could be mentioned but only a few are provided as examples. These examples (some links and sources for inquiry) are: https://courses.washington.edu/dtsclass/TEK-Menominee.pdf; https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1479546/; https://www.jstor.org/stable/10.7560/706545; Battiste, Marie. 2000. Reclaiming Indigenous Voice and Vision. UBC Press; Cook-Lynn, Elizabeth. 2007. New Indians, Old Wars. Univ. of Illinois Press; Deloria Vine Jr. 1969. Custer Died for Your Sins. Univ. of Oklahoma Press; Linda Tuhiwai 1999. Decolonization Methodologies: Research and Indigenous Peoples. Zed Books; and Deloria, Vine Jr. 1997. Red Earth, White Lies: Native Americans and the Myth of Scientific Fact. Fulcrum Press.

[4] [5] Dr. Michael Thompson led the effort for the 2015 count and is facilitating the effort for the 2016 count. Any questions please contact him at [email protected].

[6] Research and scholarship have different connotations. Therefore the word "Research" has been added by Dr. Michael Thompson to the phrase SoBI resulting in the phrase "Research and Scholarship of Broader Impacts (SoBI)" to emphasize, include, and represent the necessity of basic research and not only scholarship in terms of SoBI. See link for original SoBI meaning, https://brown.edu/go/SoBI, which is an emerging field that seeks to advance societal impacts of research through the development of its underpinnings and theoretical frameworks leading to co-creation of knowledge and knowledge transfer (Adetunji, 2016). Adding research also signifies that broader impacts conceptually is based upon a much larger more global meaning and its' applications extend past that of a research-based criterion understanding (specifically if only idealized from a modern western perspective) as well as into many other areas.

 

 

Written by:

Dr. Michael Thompson aka “The Broader Impacts Guy”

Michael is the Founding Director of the Broader Impacts in Research (BIR) organization, on the Senior Staff of the Office of the Vice President for Research (OVPR), and Affiliate Professor of Chemistry and Biochemistry at the University of Oklahoma (OU). Dr. Thompson is a member of the National Alliance for Broader Impacts (NABI). He also served on the National Alliance for Broader Impacts (NABI) Working Group, which developed the Broader Impacts Guiding Principles and Questions for National Science Foundation Proposals. For more of his posts visit, https://thebroaderimpactsguy.com/blog/.