AI Productization Profile: Education - ETS AI Labs
Personal Background Overview
Narmeen Makhani is an amazing woman with a wealth of experience with Educational Testing Service (ETS), having been involved with the organization for over 16 years. She has worked on enabling technology to roll out various large-scale assessment and learning products in K12 and higher education areas, as well as bringing innovation to digitally transform an assessment and data organization with a 75-year legacy. For the past 2.5 years, she has been leading the technology and AI team at ETS AI Labs, where she focuses on building innovative prototypes and AI capabilities that power them. She has always been at the intersection of business and technology, and her deep knowledge of ETS’ business has given her an edge in the space. Her unique skill set has allowed her to excel in the AI innovation space and led her to be a part of the founding team of ETS AI Labs.
I met Narmeen through Chief, a network of the most powerful executive women elevating women in leadership, and sat down to virtual coffee last week to pick her brains on the challenges of bringing AI innovations to full commercial productization (AI Productization).
ETS AI Labs Overview
The ETS AI Labs is the innovation center of ETS and the source of many new innovative product ideas. ETS, having pioneered automated scoring, wanted to continue to innovate with AI and leverage its potential to completely transform education in the next decade, launched the AI Labs in 2020. The AI lab was jump started from an accumulation of over 70 years of foundational research and data across K-12, Higher Education, the Workforce, and Language Learning.
The mission of ETS AI Labs is to advance educational and employment opportunities by providing tools and services that help individuals worldwide identify, efficiently progress toward, and meet their learning and career goals. They are chartered with creating research backed user – obsessed solutions that can advance the EdTech field. They focus on creating AI capabilities such as behavior prediction, authentic inference-based assessment, and actionable insights and feedback that can be productized for end users. A cross-disciplinary group of product managers, learning scientists, impact researchers, UX designers and AI technologists work together to create prototypes that can be co-designed or iteratively tested with users. When field studies have validated the new implementations, the products are transferred to the commercial business units for full commercialization.
AI Technology Areas
Multimodal AI, generative AI, speech recognition, writing quality evaluation, personalized insights
Opportunities
From Narmeen’s vantage point, ETS has unique opportunities to leverage AI and machine learning to innovate because of its accumulated research and continual interaction with users, as well as its accumulated data corpus over the decades. Models evaluating language assessments for example have trained on voice samples of non-native language learners across vast geographies and diverse language backgrounds and hence are better equipped to handle non-native speaker accents. Another area of opportunity comes from the behavior data of the users as they navigate systems. What if that data can be trained to develop models that can infer power skills like confidence, resilience, and persistence or even the simple skill of time management that is a proven predictor for success in academic life?
Narmeen is particularly interested in developing and leveraging assessments as tools for success, rather than just barriers to entry. Currently, assessments like TOEFL and GRE, offered by ETS, are primarily used to gain access to education and other opportunities. What if in addition they’re also used to help you succeed better after you gained access? Imagine that your GRE test scores also arrived with an analysis report of your strengths, weaknesses, and recommendations on how to get the best out of your graduate program that’s shared with you and your school of choice. We already know that GRE scores can be a great predictor of later success, but now they can help you attain it. Now imagine that extended across K-12, Higher Ed, and the Workforce.
Challenges
After the fun discussion of opportunities for AI innovation at ETS AI Labs, we changed topics to the challenges she faced. Until recently, operationalization of Natural Language Processing (NLP) models required intensive engineering efforts and infrastructure support to do it properly. However, the technology maturity and accessibility have drastically improved to the point where most infrastructure services are cloud-based and capabilities can be just a few API calls away. However, data labeling and annotations continue to be a critical component of building and fine tuning AI.
Another unique challenge in EdTech is bringing learning science, technology and user research together through effective product management. Great product managers at their core bridge two skill buckets: understanding the user problems and knowing how solutions can be built. With new technologies like AI and ML, there’s not a large group of product managers with experience in both, so you’re choosing to hire for one and grow the other. Hiring for experience building AI and ML solutions means you’re crossing industries with non-EdTech product managers. Hiring for user problem experience means you’re hiring EdTech product managers without AI experience. In Narmeen’s experience, the second group (EdTech experience but no AI experience) has typically been more successful than the first to grow into succeeding in creating AI and ML product solutions for EdTech at ETS. There is a uniqueness to understanding the EdTech product space, solving concurrently for user problems and optimal learning pedagogy, that create a steeper learning curve for cross-industry product managers.
领英推荐
The challenge for product management is particularly high during the blue-sky discovery stage where the target user scenario and product concept are decided on. This often requires leaders across functions to come together to envision the product because of the need for a deep understanding of user problems, value propositions, and technologies that can address them. Once a strong product concept is created and product managers and teams are aligned, teams can then take over to iterate, refine, and move through the development phase.
The other unique challenge in the EdTech space is the high barrier to achieve trust for automation. In the AI Labs, the team wanted to ensure that right out of the gate the AI they built was trustworthy, accountable, and explainable. This is done at ETS through extensive user research, strong annotation practices, measurement science and iterative testing. In the predictive applications of AI and ML models, the trust and acceptance of those predictions is the final barrier to user acceptance of the product value. It is an especially high bar to maintain for a keystone assessment provider like ETS. For AI capabilities where you can compare the prediction with actual quantifiable results, the simple accuracy percentage can provide this confidence. But if you are predicting soft skill levels with no consistent metrics, you can only benchmark against human predictions. A myriad of alpha programs, early adopter access programs and field studies are conducted for every AI capability developed to benchmark for machine/human agreement confidence levels. It’s a path of a lot of work and diligence, but crucial in establishing trust in the end user value proposition for AI predictions and decisions.
AI Productization Success
With only 2+ years under their belt, the ETS AI Labs has had some early successes in building innovative AI capabilities such as the use of multimodal AI to make inferences and automating the generation of assessment content. Multi-modal AI involves incorporating data across parameters of speech, text, body language, facial recognition, and user behavior to provide feedback, to diagnose, and to predict skills.?The data source for the model training spans internal data as well as external sources.
Generative AI is a hot topic today due to ChatGPT, but AI Labs have been very hyper focused on context specific and personalized generation for education and work force scenarios. Imagine an interview practice app that can drill down on your experiences with solving collaboration problems to help you practice for your interview, give you recommendations, and diagnose areas to improve. This is a capability that moves assessments from barrier to opportunity access to conduit to success.
Narmeen is a firm believer that it will take a coalition and collaboration across institutions and organization to solve education access inequality through leveraging AI and the ETS AI Labs is open to partnerships and collaboration.
CTO | Advisor to corporates & B2B midmarket on TRIUMPH transformation for profitable growth | Speaker | Author | Board Member | Innovation | Strategy | Change Management | Chief Transformation Officer
1 年I don't think people appreciate how AI permeated every aspect of our lives, including innovation. Thank you for this series Yuying Chen-Wynn
Co-founder & COO of HeyHi (SmartJen)
1 年Resonated most with the challenges on data labelling and annotation! I would like to add on to digitisation and tagging of content, particularly for assessment based system which the recommendation system relies on classification model to recommend content based on the learning gaps. How to make it easy, and seamless, to digitise + categorise learning content with ML models, remain the top problem to solve in my team - crucial step to adaptive assessment and adaptive learning. While there are already APIs ready to be called, however, when it comes to high degree of granularity specific for a particular curriculum, those models are usually not good enough and that means, customisation is needed. That adds up to the challenge list.
Head of AI at PEAK6
1 年Narmeen Makhani