Need for a DARPA for Education in 2021

Summary

COVID-19 has revealed the inadequacies of current distance learning approaches, especially education technologies (EdTech).  These same EdTech applications are desperately needed in the classroom as well. Like many societal upheavals, the pandemic allows us to look at problems from new perspectives and seek new approaches to solving vexing problems, especially in education. The Biden administration should create a rapid-cycle Advanced Education Research Program (AERP) to accelerate highly effective education technologies (EdTech), especially developing advanced digital tutors that meet or exceed human tutors' capabilities.  AERP would use the same iterative research and development techniques as the Defense Advanced Research Projects Agency (DARPA) that have produced some of society’s greatest innovations such as the internet, new vaccine development methods, and advances artificial intelligence.[1] An AERP would help us reconceptualize learning and solve some of the most vexing problems in education by identifying promising “moonshots” and then creating multidisciplinary teams to achieve them.

An AERP is needed to move the needle on student performance across the academic spectrum. However, we are proposing high school mathematics as the first target of this program with a North Star of creating learning architectures that have an awareness of a student’s strengths and weaknesses across academic disciplines and collapse arbitrary distinctions between STEM subjects. The immediate goal would be to create tools that help failing students reach proficiency. The 2019 National Assessment of Educational Progress (NAEP) scores for 12th grade indicate that only 37% of graduating seniors met or exceeded college requirements in reading or mathematics.[2]  This unacceptably low overall achievement level is statistically unchanged since 2015. However, it hides the disturbing trend that lower and middle-performing students have seen a consistent decline in performance.  As with the digital divide, the overall educational divide is widening, especially among Black and Native American students. The abrupt transition to online education during COVID-19 is widening these gaps.[3]  

We can do better! In 1984, Benjamin Bloom reported that individual tutoring would enable students to perform better than 98% of their classmates in traditional school settings, a two standard deviation improvement in performance.  However, 1:1 human tutoring is not scalable and is known as the 2-sigma problem[4]. Why has this not been achieved? One answer is that the development process used by industry and most of the federal government is not conducive to rapid development and innovation. This bold challenge is precisely the sort of problem the DARPA model will solve. AERP would tackle the 2-Sigma Problem, starting in mathematics, especially for underserved students in both urban and rural environments. Research has shown that no single approach can reach the level of improvement we are seeking[5]. Success will require designing and developing EdTech in fundamentally different ways. AERP would fund interdisciplinary programs that would bring together top experts in education, research, and advanced technology to develop highly effective digital tutors as engaging as the best video games and combining different education approaches.  The goal would not be to replace teachers but to provide the tools they need to understand and meet students' unique needs. Beyond math, AERP would promote real-time formative assessment that would dramatically reduce the need for formalized assessments and testing that take time away from learning.  Real-time formative assessment will allow teachers to intervene immediately during the learning process to help students as they are learning instead of when it is too late.  For all AERP initiatives, scaling will be paramount so that these tools reach students most in need in both urban and rural environments.  Products from the program will be open-source and transition directly to the classroom or as tools to other EdTech developers.

Challenge & Solution

Although EdTech promises to close education gaps and provide highly individualized instruction for all children and adults, it has not lived up to this promise, primarily because of the way it is being developed and transitioned to the classroom.  Given this structure, few programs have significantly impacted their target audience, and even fewer have been able to scale. Technology research and development are costly, and profit margins for education products are low, so education companies rarely invest in the research needed to develop innovative, evidence-based products for the marketplace.  Philanthropies currently have some promising efforts, but the funding level is far below the transformational levels required to meet the Nation's needs.  The federal government is uniquely positioned to provide the investment and national coordination needed for rapid progress if it creates programs using the DARPA model rather than traditional funding models.

The current research model most commonly employed in education across public and private funding organizations (including philanthropies) was developed just after World War II and became the template for most federally funded research programs in the United States. The approach is best summarized in Vannevar Bush’s seminal report in 1945, “Science the Endless Frontier.” It promotes the commonly used distinction between basic and applied research.[6] It has been a very successful model, but it is a single research strategy. It may not always be the most optimal path for solving problems in education, especially EdTech. In the behavioral sciences, including education research, there are often disconnects between basic and applied research due to the very noisy and context-specific nature of human data. Applications often fail due to a lack of generalizability of basic research findings.  Also, research funding provided in typical research programs is not adequate to produce high-quality research and a high-quality product.

Using these traditional research approaches has had mixed success.  Applications have primarily failed to scale to racially diverse populations of students in urban and rural areas. Ideally, technology should be an aid and a force multiplier for teachers to achieve the benefits of providing 1:1 tutoring for students, even in a large classroom or virtual classroom setting. Current technology solutions are often perceived to impede educators and are not adequately integrated into the learning environment. What is needed is a process that takes a holistic approach to development. AERP will create products with the student, teacher, environment, and infrastructure issues considered throughout the research and development process.

DARPA programs are structured to conduct basic research towards an applied goal and receive adequate funding to achieve the ultimate goal. Donald Stokes best articulated this strategy in his book, “Pasteur’s Quadrant” in 1997.[7] The role of Pasteur’s Quadrant at DARPA and its potential impact on education research has been written about in much greater detail. However, in brief, basic and applied research are collapsed into an iterative process aimed at solving specific challenging problems. The result is a program structure that accepts higher technical risk at the outset but fosters rapid innovation by minimizing the disconnect between basic and applied research programs. Because basic and applied research is conducted in tandem, initial program funding needs to be higher. However, if successful, returns on investment can be considerable, and the development cycle considerably shorter.  Programs should be funded at a level that produces both high-quality research and products that are ready or nearly ready for a transition. Results of successful programs do not languish. 

Because of the innovation model and funding levels, using the DARPA model would allow setting bold education goals and rapidly iterating towards products to transition directly to the classroom. Alternately, tools and techniques will be transitioned to the education industry.  DARPA identifies ambitious goals, or “moonshots,” and creates programs to reach them in 3-4 years.  As challenges are identified during development, research is conducted that feeds back into the ultimate product.[8]  

Promising Examples

There are some excellent examples of the DARPA process successfully applied to education. The $35M DARPA ENGAGE program was a 4-year program aimed at developing STEM educational games that adapted to the individual student and evolved its teaching strategy continuously based on the thousands of players using the platform.[9]  The DARPA Education Dominance Program was a more focused $50M+ program that produced an advanced digital tutor for Navy technical training.  The system produced is reported to have far exceeded the existing classroom experience results and has claimed to be on par with 1:1 tutoring.[10] Both programs produced specific products and general capabilities and techniques that have been used in subsequent applications.   As part of this program, DARPA worked closely with the U.S. Dept of Education and co-sponsored a 2012 joint Small Business Innovation Research (SBIR) Program demonstrating the compatibility of the DARPA model to education.[11]

An example program is currently being funded jointly by the Chan Zuckerberg Initiative and the Gates Foundation, which have initiated a 5-year program to develop tools to blend math skills with executive functioning.  The theory is that this combination will improve both. First-year funding for this program was reportedly around $9M.[12]  However, a federally funded model would work toward 3-4-year programs and take on focused and broadly scoped initiatives.

Like these earlier programs, AERP envisions a reconceptualization of the use of education technologies.  What are some possibilities? Instead of dry tutoring systems, architectures could combine tutors, games, video, chatbots, and other formats.  Rather than developing tools with only a single topic in mind, one could create tools that support students through entire curricula.  Tools could be holistic educational applications that focus on the whole student, harnessing theories and best practices from neuroscience, intelligent tutoring, executive function, social-emotional learning, psychology, and creativity.  The research could explore how to effectively blend active learning in online environments.  Instead of exams, education outcomes could be guided by real-time formative assessment. The tools will learn to know the student, understanding academic strengths and weaknesses, personal preferences, attention patterns, and even social-emotional factors. Systems will then adapt instruction accordingly.  For instance, one approach to mathematics instruction would be to give students example problems based on their interests in science, sports, or other areas.  Suppose a student shows aptitude and interest in a specific area. In that case, the system will attempt to present future information in formats that will interest the student.  Architectures would be modular. The content and approaches would be crowd-sourced so that the system continues to improve over time. Systems would provide dashboards to the teacher with intuitive, advanced tools to provide feedback on student progress.  AERP will ask what we can do with technology that we cannot do in the traditional classroom.    

Proposed Action

Creating an AERP would require active support from the White House and the Secretary of Education, federal budget commitments, and coordination across federal and nonfederal entities that support education. Program goals would be created and refined with the help of interdisciplinary panels of top experts.

The Secretary of Education would appoint an Executive Director for AERP who would work with the White House Office of Science and Technology Policy (OSTP) to convene experts to include classroom teachers, education researchers, philanthropies, and industry to identify and refine moonshot level challenges.  These panels would also identify potential paths for the transition.  Once promising programs are identified, the Executive Director would recruit program officers who are world-class experts in the areas of interest.

DARPA-style programs tend to be higher-risk and require a unique, nimble program management approach. Projects within a program are not on a standard fixed schedule as they are created and terminated based on the program's needs and what is learned along the way. 

AERP programs would be structured to fund programs in the $25M to $50M range on a 3–4-year fixed schedule. Program managers would be selected from top experts in the education field for a 3-4-year term and build interdisciplinary teams of experts to solve the identified problem, including experts from outside education who may lend unique and innovative expertise.  To optimize success, teams would be selected that would include teachers, administrators, developers, and top scientists to gain insights into the problem and the feasibility of the solutions. Like DARPA, program statements would be defined using Heilmeiers Catechism[13], a set of simplifying questions that bring structure to a developing program idea. These questions {modified slightly} include:

• What problem are you trying to solve? Articulate using absolutely no jargon. 
• How is it done today, and what are the limits of current practice? 
• What is new in your approach, and why do you think it will be successful? 
•  Who cares? If you are successful, what difference will it make? 
• What are the technical risks?
• How much will it cost? How long will it take? 
• What are the mid-term and final “exams” to check for success?

All programs will be aimed at creating high-impact, scalable digital tutors aimed at the identified problems. Key to this innovation and scalability would be a close relationship of the research programs with actual education environments (physical or virtual) representative of a wide array of students. The iterative development process would continuously cycle between developers, researchers, and learning environments. The approach would also be flexible by necessity. As things are learned during the iterative process, new research projects are created as needed, and projects that do not contribute to the solution are terminated. Program goals can change as basic assumptions are challenged, and knowledge is gained.

Conclusion

Now is the time to create and fund this endeavor. The concept of a DARPA for education has been in the works for almost a decade. COVID-19 and NAEP scores have highlighted the need for new and innovative education approaches, especially education technology.  The AERP model provides an alternate research and development pathway that has been highly successful in many areas, including the few areas in education where it has been applied.   We need to create this vehicle for innovation soonest so that we can eliminate the educational gaps for students at most risk, especially in rural and urban communities.  Applying this process to education will help us meet students' needs to prepare them for college and career.  Finally, we need to be prepared for the next pandemic or national emergency to keep our students at home and away from the physical classroom.

About the Author

Russell Shilling was the Executive Director of STEM at the U.S. Department of Education during the Obama Administration, overseeing the Department’s policies to drive innovation in STEM education and enhance interagency coordination. He served as the Chief Scientific Officer for the American Psychological Association, acting as a policy advocate for research, education, psychological health, and various other issues where behavioral sciences play a significant role.  Previously, Shilling served as a Navy Captain, retiring after 22 years of service as a Navy Aerospace Experimental Psychologist focusing on education, training, and psychological health. Shilling held various leadership positions, including program management positions at the Office of Naval Research (ONR) and the Defense Advanced Research Projects Agency (DARPA). He has served as an Associate Professor at both the United States Air Force Academy and the Naval Postgraduate School. Shilling was an early pioneer in the “serious” games movement and was responsible for award-winning games for STEM education, medical training, and psychological health. He pioneered research to treat post-traumatic stress with virtual reality and graphic novel storytelling tools and developed programs with Sesame Street Workshop that have helped military children adapt to traumas caused by deployments, injury, and grief. These programs have won numerous awards, including Emmy nominations and Parents Choice Awards. Shilling also advocates for improved resources for autistic students and adults.

References

[1] Russell Shilling, “Why We Need a DARPA for Education, Scientific American”, 2015, https://www.scientificamerican.com/author/russell-shilling/

[2] National Center for Education Statistics, “The Nations Report Card”, 2019, https://www.nationsreportcard.gov

[3] World Economic Forum, “COVID-19 is Widening the Education Gap. This is How We Can Stop It”, 2020, https://www.weforum.org/agenda/2020/08/we-must-resist-covid-19-widening-us-education-gap/

[4] Benjamin Bloom, “The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One to One Tutoring”, 1984, https://doi.org/10.3102%2F0013189X013006004

[5] Hattie, J. A. C., “Visible Learning: A synthesis of over 800 meta-analyses relating to achievement”, 2009,https://link.springer.com/article/10.1007/s11159-011-9198-8

[6] Vannevar Bush, “Science The Endless Frontier”, 1945, https://www.nsf.gov/od/lpa/nsf50/vbush1945.htm

[7] Donald Stokes, “Pasteur’s Quadrant: Basic Science and Technological Innovation”, Brookings Institution Press, (1997). 

[8] Defense Advanced Research Projects Agency, “About DARPA”, 2020, https://www.darpa.mil/about-us/about-darpa

[9] Defense Advanced Research Projects Agency, “ENGAGE Program”, https://www.darpa.mil/program/engage

[10] Dexter Fletcher, “Accelerating Development of Expertise: A Digital Tutor for Navy Technical Training”, 2015, https://apps.dtic.mil/dtic/tr/fulltext/u2/1002362.pdf

[11] Institute for Education Sciences, “Joint-Agency Solicitation for Education Technology Games”, 2012, https://ies.ed.gov/sbir/pdf/GamesPhaseI2013.pdf

[12] Linda Jacobson, “New Grant Program Will Focus on Executive Function, Math”, Education Dive, 2018, https://www.educationdive.com/news/new-grant-program-will-focus-on-executive-function-math/561405/

[13] George Heilmeier, “Heilmeier’s Catechism”, https://www.darpa.mil/work-with-us/heilmeier-catechism