Artificial Intelligence #27 AI trends in research

Firstly, this week, we crossed 30K subscribers (in less than 6 months). Thanks for your support. In a world of short content formats like twitter and Instagram, its nice to see that this longer format has a following also

As I mentioned last week, in Nov and Dec, I start teaching the two courses at the University of Oxford -?Digital Twins: Enhancing Model-based Design with AR, VR and MR , and?Artificial Intelligence: Cloud and Edge Implementations .

Just before my teaching, every year, I always track the state of ai report . This is an extensive report which gives a set of good markers for me to validate my ideas. The report comes with some caveats (ex a specific set of contributors) and I suspect also fields like healthcare which are of interest to the authors. But overall, it’s a good view point for complex AI research trends. Below are things from this report that I found interesting. Many of these under currents we have been tracking before – for example Pytorch, transformers etc.

Some are worth noting for the traction getting in mainstream industry ex

• Real-time computer vision protects employees from workplace injuries (or worse) Intenseye
• Reinforcement learning for an effective Covid testing strategy (in Greece)
• Viz.ai’s stroke detection software helps 1 patient every 47 seconds in the US today

?While others solve very real problems in the industry

• Careful data selection saves time and money by mitigating the pains of big data:

?Key themes in the 2021 Report include:

• AI is stepping up in more concrete ways, including being applied to mission critical infrastructure like national electric grids and automated supermarket warehousing optimization during pandemics.
• AI-first approaches have taken biology by storm?with faster simulations of humans’ cellular machinery (proteins and RNA). This has the potential to transform drug discovery and healthcare.
• Transformers have emerged as a general purpose architecture for machine learning, beating the state of the art in many domains including NLP, computer vision, and even protein structure prediction.
• Investors have taken notice, with?record funding this year into AI startups, and two first ever IPOs for AI-first drug discovery companies, as well as blockbuster IPOs for data infrastructure and cybersecurity companies that help enterprises retool for the AI-first era.
• AI is now an actual arms race rather than a figurative one.?AI researchers have traditionally seen the AI arms race as a figurative one -- simulated dogfights between competing AI systems carried out in labs -- but that is changing with reports of recent use of autonomous weapons by various militaries.

Below is the list of topics form this report on my radar (focussing mainly on AI research)

• Attention-based neural networks move from NLP to computer vision in?achieving state of the art results.”
• Self-supervision is taking over computer vision: Facebook AI introduces SEER, a 1.3B parameter self-supervised model pre-trained on 1B Instagram images that achieves 84.2% top-1 accuracy on ImageNet, comfortably surpassing all existing self-supervised models.
• Transformers take over other major AI applications, e.g. audio and 3D point clouds Self-attention is the basic building block of SOTA models on speech recognition...
• Games continue to drive Reinforcement Learning research MuZero is the latest member of DeepMind’s “Zero” family. It matches AlphaZero’s performance on Go, chess and Shogi, and outperforms all existing models on the Atari benchmark while learning solely within a world model. Muzero appeared in Nature in December 2020.
• Zero-shot generalisation in reinforcement learning RL agents have shown impressive performance on challenging individual tasks. But can they generalize to tasks they never trained on? DeepMind trained RL agents on 3.4M tasks across a diverse set of 700k games in a 3D simulated environment, and show they can generalize to radically different games without additional training.
• GANs have a serious new adversary: diffusion models Diffusion models’ training is more stable than GAN’s and outperforms them on several well-established datasets in image generation, audio synthesis, shape generation and music generation.
• 26% of AI research papers make their code available and 60% make use of PyTorch
• Deep generative models offer highly accurate probabilistic predictions of precipitation Predicting rainfall at high-resolution with a short lead time (<2h, i.e. “nowcasting”) is important for businesses and people when making weather-dependent decisions. New deep generative model (DGM)-based methods bring added resolution and prediction accuracy beyond that of physics-based simulations and current ML methods.
• Here comes a new framework challenger: JAX Introduced by Google in late 2019, JAX is a python package that combines Autograd (a library for automatic differentiation) and XLA (a compiler for linear algebra) to accelerate computations for machine learning research.
• Careful data selection saves time and money by mitigating the pains of big data: Working with massive datasets is cumbersome and expensive. Carefully selecting examples mitigates the pain of big data by focusing resources on the most valuable examples, but classical methods often become intractable at-scale. Recent approaches address these computational costs, enabling data selection on modern datasets. Based on paper SEALS: Similarity Search for Efficient Active Learning and Search of Rare Concepts
• Real-time computer vision protects employees from workplace injuries (or worse) Intenseye’s computer vision models are trained to detect over 35 types of employee health and safety (EHS) incidents that human EHS inspectors cannot possibly see in real-time. The system is live across over 15 countries and 30 cities, having already detected over 1.8M unsafe acts in 18 months.
• Computer vision unlocks faster recovery from natural disasters: Climate change is increasing the severity of natural disasters, inflicting $190B of damage to homes worldwide in 2020, 4x more than in 1990. The global population exposed to natural disasters will increase 8x by 2080. Tractable's AI-augmented system allows homeowners to take photos of damage to their home after a natural disaster (e.g. hurricanes) to predict repair costs and unlock insurance claim payouts months faster.
• UK National Grid ESO halves error of electricity demand forecast using transformers Predicting demand is essential to achieving ESO’s ambition of running the grid on net-zero generation by 2025.
• Reinforcement learning for an effective Covid testing strategy: One of the few real-world deployments of AI that addresses the pandemic is the reinforcement learning (RL) system, Eva, which was developed in Greece. Given a specified fraction of travellers who could be tested, Eva selected which specific passengers to test at the Greek borders. Eva identified 1.5x - 4x more positive infections at a given testing fraction than random selection.
• Viz.ai’s stroke detection software helps 1 patient every 47 seconds in the US today A stroke occurs when the brain is deprived of its blood supply. Within minutes, brain cells begin to die from a lack of oxygen and nutrients, which results in irrecoverable damage. Rapid detection of brain strokes is crucial, but clinically challenging. In 2021, a real-world multi-center study of 45 stroke patients tested a deep learning system from Viz.ai versus standard of care. It found that the AI-based approach reduced the transfer time for a patient post-imaging at a primary stroke center to a comprehensive stroke center by 39% on average.
• Public market investors favor AI-first cybersecurity players: CrowdStrike ($60B), Darktrace (￡5B), SentinelOne ($18B), Riskified ($6B) - In the last 12 months, CrowdStrike has almost doubled its market capitalisation to $60B and reached $1.3B ARR. The company is demonstrating the platform potential of AI-first technology in cybersecurity: 53% of its 13,080 subscription customers purchase more than 5 products and 29% subscribe to more than 6 products. Meanwhile, SentinelOne (124%) and CrowdStrike (120%) are firmly in the high-growth net dollar retention segment of SaaS companies, suggesting that their customers expand their subscription spend year on year.
• The enterprise data and automation sector is on fire: Snowflake, UiPath, Confluent IPOs UiPath (robotic process automation), Snowflake (cloud data platform), and Confluent (Kafka-based data streaming) represent $138B of newly created public market value in 2021 with revenues growing 50-100% YoY at this scale. All three companies have best-in-class net dollar retention above 130% and 2% of their customer base spending over $1M per year. Snowflake became the largest software IPO of 2020, raising $3.35B.
• Databricks: The enterprise data/AI juggernaut reaches $38B valuation and $600M ARR. Since launching its original data platform built on Apache Spark in 2015, Databricks has grown into a one-stop home for (un)structured data, automated ETL, collaborative data science notebooks, business intelligence using SQL, and full-stack machine learning built on open source MLflow.
• Google infuses AI capabilities into more of its business and consumer applications
• Large language models for all: startups raise $375M to translate research into industry Startups in the US, Canada, and Europe raise close to $375M in the last 12 months to bring large language model APIs and vertical software solutions to customers who cannot afford to directly compete with Big Tech.
• Deep reinforcement learning-enhanced picking robots support a surge in online grocery Robotic picking and packing is helping retailers meet a growing demand for online deliveries. Leading online grocery technology company, Ocado, uses computer vision and proprietary grasping technology to efficiently pick and pack items for grocery orders. In e-commerce, robotic picking platform SORT will handle 300M+ items by the end of 2021. Reinforcement learning tool (RLScan) is a very early example of RL success in production environments of robotic systems at scale.
• Deep learning automates 98% of stock replenishment decisions for online grocers
• Browser-based federated learning thrives in a post-cookie world: With the regulation of third party cookies and the increasing public awareness of the importance of data privacy, browsers are compelled to find new privacy-preserving solutions for their advertising business.
• AI Ethics: Timnit Gebru’s firing from Google shocks the AI community - Dr Gebru left Google after a substantial disagreement over a research paper which examined the risks of large language models, including bias and the carbon footprint associated with training these models.
• AI Governance: enter Anthropic as a potential third pole for AGI research Many of OpenAI’s leading researchers leave to start a major new AI research lab.
• The EU continues to be the first (and heavy handed) mover in AI regulation - The EU introduced a proposal for AI regulation (AI Act) in April 2021. The proposal aims to provide the necessary legal certainty to facilitate innovation while ensuring the protection of consumer rights. Like GDPR, the proposed law concerns any person or organization, even foreign, involved with an AI system placed or used in the EU. But the AI Act goes beyond GDPR by aiming to directly regulate the use of AI systems.
• Military AI: Anduril continues to gain momentum
• Military AI: Microsoft’s huge $22B contract for Hololens moves them closer to a defense prime.

?This week, I also shared my views on the slingshot blog with Dr David McKee who is on the course representing the Digital Twin Consortium ?

If you want to learn with me at #universityofoxford see

-?Digital Twins: Enhancing Model-based Design with AR, VR and MR , and?Artificial Intelligence: Cloud and Edge Implementations .

?Image source Roland Pippes – pixabay