Use Cases for Foundation Models (aka LLMs) in the Energy Industry

Foundation Models as a Superset of Large Language Models (LLMs)

Recently, Stanford Center for Research on Foundation Models released a report on foundation models where they define them as models trained on broad data that can be adapted to a wide range of downstream tasks. These models demonstrate surprising emergent capabilities and substantially improve performance on a wide range of downstream tasks. Similar to humans, they can perform a lot of things which they haven't been "trained" on yet.

Foundation models are the subject of a growing paradigm shift, where move from a collection of "vertical" AI models that are very specific to certain problems and needs a lot of maintenance (retraining) to a very few "horizontal" foundation models?that are repeatedly reused as the basis for many applications.

Foundation models are a superset of LLMs, although the most "well known" foundation models currently are LLMs (e.g., GPT-3). "Foundation model” emphasizes the function of these models as foundations for downstream applications, whereas “large language model” emphasizes the manner in which these artifacts are produced (i.e., large textual corpora, large model size, “language modeling” objectives). Analogous to how deep learning was popularized in computer vision but now extends beyond, foundation models emerged in NLP with LLMs but foundation models (that are not LLMs) exist for many other modalities, including images, code, proteins, speech, molecules as well as multimodal models. In this article, I will use the term foundation model to explain the more generalized nature of the approach.

Use Cases for Foundation Models in Energy Industry

Generate

• Generate text from data, similar to an X-Ray specialist to provide a written document generated from the ultrasound images, we can write human readable reports from the data in dashboards and databases. In CCS operations, there are a lot of documents that needs to be presented to the regulatory authorities, which could be automatically generated from the structured data.
• Generate synthetic data for simulation, AI model training and testing. More data makes the AI models better, foundation models are a great way to generate data based on a small set of examples. Also useful when it is not possible to utilize real world data due to confidentiality reasons. Data would also be useful to test value chains where a lot of connected systems needs to work together, such system of systems could be easily tested.

Summarize

Data platforms, such as OSDU, works like a library, where basic information on very large data files of differing formats are extracted and indexed. Similar to finding a book, it is only searchable by the metadata and searching on the details is only possible once the whole file is downloaded and opened by specialized applications. The findability (if this is a proper word) is determined by how rich the metadata is.

• Extract information from reports, emails, etc… into summaries which could be indexed. Summary of the document could be automatically extracted during ingestion and could be part of the metadata that is indexed that will significantly boost the findability.
• Extract metadata, keywords from data for generating metadata automatically. With the foundation models we can automate and enhance the named entity extraction of metadata from various data types and document. Ingestion into platforms such as OSDU will be automated.
• Summarize legal documents such as ESG reports, etc… In a reverse way, important information could be extracted from legal reports and documents as structured data that could be ingested into databases.

Rewrite

• Rewrite automated meeting transcriptions to correct errors and language mistakes. A second use case would be to generate grammatically corrected documents from written and voice notes taken by the field technicians. This way we will save the time for personnel that is spent in writing documents and reports.
• Redact personally identifiable information or sensitive data. Documents could be sanitized before being shared or ingested into data platform. Same could be used to sanitize documents for sensitive data before used in training AI models.
• Turn a complex text into simple language, legal or highly technical jargon could be simplified before being shared with a general audience. Document style could be converted automatically to address specific reader personas.

Extract

• Extract named entities from a document which could be converted into database schemas. For example a document that describes how carbon emissions should be reported could be used to generate the set of data entities and attributes of each entity.
• Extract metadata keywords, tags, similarly the named entities, additional information could be extracted to enrich metadata while ingesting into data platforms.

Search

• Advanced GPT like search on OSDU documents. Currently search is limited to metadata indexes, it would be possible to do deep indexing and searches on the actual content of the data and documents.
• It is also possible to enrich the metadata for simplified search queries. From metadata only simple keyword search to deep document search.
• Query structured data using Natural Language. What if you “ask” your datasets questions in natural language? And what if you ask not only to compute numerical output but also generate analysis about trends, KPIs, and best next actions?

Cluster

• Organize documents and data into buckets before extracting and ingesting into data platforms. Data and documents are spread into variety of locations and very hard to determine the type and content.
• Contextualize data by associating similar document / data. Once data is ingested into a data platform such as OSDU, similar data could be clustered across domains, fields, types, etc... and can be fused together to provide 360 view of assets.

Classify

• Arrange documents and data into groups which could be processed and ingested in similar way.
• Assign documents and data to OSDU kinds during the ingestion to automate the quality checks and metadata extraction.

Chatbots

• Chat GPT being the most widely known usage for LLMs, same could be achieved for domain specific data in OSDU. Instead of limiting the search with specific metadata fields and search queries, one can "speak" to the data and search using natural language. The bots will return answers to questions rather then a list of documents and data that has the answer and leaving it to the user to extract the answer.

Translate

• Translate between human languages. International companies have presence across the world, documents in OSDU could be automatically translated to multiple languages during ingestion. Translation could also be provided on demand during document extraction.
• Translate between computer languages. AI could be used to translate between different languages and frameworks such as Bash, Perl, Python, etc... When trained with the right input is could also be used to translate to/from generalized to domain specific languages.

Codex

• Write code automatically, it is possible to write code that ingests, searches and extracts data from OSDU using various languages and frameworks. As an extension to zero code / no code platforms, it is possible to build complex applications with minimal development background.
• Write automation scripts, automation of administrative tasks that are related with OSDU such as adding new users, giving access, creating new data partitions etc... are all API driven. Furthermore all LOB services could be scripted to automate the tasks of loading data, subscribing to changes etc... Platform administrators could create automation tasks easily.
• Write step by step instructions for fixing mechanical issues or activities. An extended use case would be definition of pseudo languages and writing step by step scripts for the field personnel to fix or change equipment configurations. These could be automatically generated as part of the predictive maintenance operations.

Image Generation

• Automated generation of images for step by step maintenance or installation instructions. Generation of images that are part of the step by step instructions generated for the maintenance would simplify the job of the field personnel, especially if they're visual learners. Foundation models such as Dall-E are becoming very successful for this task. As the technology matures, videos could also be generated instead, Meta's Make-a-Video models provide very promising results. Lastly, not specifically image generation, but generating audio instruction would also be possible with the foundation models such as Microsoft's Vall-E.
• Generate synthetic test data for training AI models for computer vision tasks. One of the key problems with training video recognition is the lack of training images, foundation models such as Dall-E could be utilized to generate training images for industrial applications such as automated inspection systems that can detect and report anomalies by visual inspection of equipment and sensors.

OSDU as an Enabler of Foundation Models

My colleague Einar Landre has written an amazing article that summarizes OSDU which could be found here https://lnkd.in/dQGVpnxM .

In short, OSDU is a library (which I have to give the credit to my colleague Graham Cain ) where it Clusters and Categorizes data (books), Extracts metadata for indexing (think of library index cabinets, which shows my age it is all online now), in some cases Summarizes for quick abstract searches and puts the data in generalized storage for easy access (think of books in shelves). One can Search the indexes, finds the files and can get access to the actual data. (ask the librarian to bring the book)

Familiar huh, in the previous section I have explained the use cases where Industry Data Platforms such as OSDU would utilize foundation models to further enhance the user experience. However, if we also look into the flipside, as you see in above paragraph which explains the key functions of OSDU, the items in bold are very similar to what foundation models deliver and hence OSDU could be a great enabler for Foundation AI Models, aka LLMs, and the capabilities they provide. OSDU gets all the Energy World's data, puts it in a common place and serves it via computer consumable APIs. With the current design of OSDU as a System of Record, these APIs are provided as https rest endpoints, which is not arguably the best option for high performance AI workloads such as training Foundation Models (aka LLMs). One of our colleagues Markus Cozowicz have released a Spark connector for OSDU https://github.com/microsoft/OSDU-Spark and Eirik Haughom have built a solution (https://github.com/EirikHaughom/MicrosoftEnergyDataServices/tree/main/Guides/Synapse/DataLakeIngestion) to ?automatically ingest binary data created on an external datastore into Microsoft Energy Data Services, which is Microsoft's implementation of OSDU on Azure (see last section for more information).

Although there are band-aid type solutions as outlined above, the ultimate goal is to build an AI Consumption Zone for OSDU where the data that resides in the data lake which holds all data files (remember the library analogy, these are all the shelves of the library) and exposes them as AI consumable structures. Some data files formats such as Wellbore DDMS that stores the data in Parquet format are easily consumable as external files to modern data warehouses. Other data formats such as OpenVDS also holds the promise to be accessible as AI consumable data, its founder Bluware has already demonstrated AI capabilities on SegY formats. Other formats needs to be evaluated for suitability for direct AI consumption to train Foundation Models (aka LLMs). My personal criteria here would be "could we expose the file as an external table in a modern data warehouse"? Work yet needs to be done, however there's good progress and stay tuned.

Microsoft Energy Data Services

Microsoft Energy Data Services is an enterprise-grade, fully managed, OSDU Data Platform for the energy industry that is efficient, standardized, easy to deploy, and scalable for data management—for ingesting, aggregating, storing, searching, and retrieving data. The platform will provide the scale, security, privacy, and compliance expected by our enterprise customers.

Learn more

• Get started with Microsoft Energy Data Services today.