Using generative AI to close knowledge gaps across domains: jargon 1 + jargon 2... + llm = english

What follows is a relatively detailed example of one way I've come to depend upon LLM AI tools to do my daily work.

I'm trying to better understand the challenges of renewable energy projects in the power sector, an area where Stride has been working as a software consulting business for the last two years and where I believe we can focus our growing expertise to provide profit to our business and a positive impact in the world.

One of the pain points I hear from people is the difficulty keeping track of and understanding the regulatory environment that determines so much of what Utilities do and forms the pathways by which Distributed Energy Projects can or cannot find a path to viability.

So, I did a quick experiment to begin to validate whether this was a problem that could be helped by the use of Large Language Models.

First I had to find a relevant issue in the regulatory scheme. I took the September Federal Energy Regulatory Commission (FERC) meeting agenda and the fact I reside within the New York Independent System Operators (NYISO) jurisdiction to focus a test.

FERC issues agenda documents short enough to paste into a ChatGPT prompt.

NOTE: This example contains all public documents. I can use ChatGPT without fear of privacy concerns in this case. I used the paid tier (have access to plugins) and I've submitted the opt out request.

"Summarize for me, not a domain expert in the power sector, which of these FERC meeting agenda items include consideration of the NYISO and what is being investigated" <paste in agenda items>

In the upcoming FERC meeting, the agenda items that concern NYISO are under E-3. FERC initiated investigations into NYISO's existing open access transmission tariffs (OATTs) to determine if they are unjust, unreasonable, or discriminatory. Two specific elements are under review: mark-to-auction mechanisms for financial transmission right (FTR) market participants' collateral requirements and volumetric minimum collateral requirements for FTR market participants. NYISO filed its response to this Show Cause Order on October 26, 2022. Agenda item E-3 will likely be an order based on NYISO’s response.

Looking at the agenda document, this is related to FERC Order EL22-64-000

Searches on the NYISO's filings database and FERC's own eLibrary showed that the only significant public document on this order is NYISO's filing in response. The legal document itself is a 25 page PDF. It is dense with Power Sector, Regulatory, and Legal domain specific language. It is this particular collision of multiple areas of specialty that makes understanding these documents so difficult.

Using a freemium ChatGPT plugin that uses a single PDF as grounding, I enabled and followed the plugin's instructions and was given this document summary.

FERC Order EL22-64 questioned the New York Independent System Operator's (NYISO) credit requirements for Transmission Congestion Contracts (TCCs). Specifically, the Commission asked whether NYISO should include a volumetric minimum collateral requirement to adequately address the risk of default associated with TCCs. The Commission directed NYISO to either justify its existing tariff as just and reasonable or propose changes to remedy the Commission's concerns.

In response, NYISO argued that its existing credit requirements for TCCs are robust and appropriately calibrated to reflect the associated default risk. NYISO's credit policies have been tailored to minimize default risk while supporting market liquidity. They require TCC bidders and holders to provide collateral in amounts that reflect the default risk throughout the term of the TCC. NYISO also conducted analyses demonstrating the effectiveness of its credit requirements, even in scenarios similar to the GreenHat default, which was cited by the Commission. NYISO concluded that its existing credit requirements remain reasonable and commensurate with the risk involved.

The information was found on pages 1, 2, 8, 12, 13, and 16 of the document.

Yup. That's a summary. Can't say it is jargon free...

"Please explain the last response in terms someone not conversant with legal or power sector concepts."

Certainly. The Federal Energy Regulatory Commission (FERC) questioned how a New York energy organization (NYISO) manages the financial risks associated with certain energy contracts. These contracts, known as Transmission Congestion Contracts (TCCs), help manage the flow of electricity across the grid. FERC wanted to know if NYISO had enough financial safeguards in place to handle situations where parties might not be able to fulfill their contract obligations.

NYISO responded by saying that they've got it covered. They have strong rules in place that require anyone entering into these contracts to put up enough money as collateral. This ensures that if someone can't fulfill their part of the deal, the financial impact is minimized. NYISO also said they've run tests to prove that their system works, even in tough situations like a big company defaulting on its obligations.

In short, FERC asked NYISO if they were doing enough to manage financial risks in the energy market, and NYISO said yes, backing it up with data and analysis.

Besides dumbing things down a bit too much. This summary doesn't connect with the initial description of FERC's concerns for unfairness in Open Access Transmission Tariffs.

"What is the relationship between TCC's and OATTs"

Transmission Congestion Contracts (TCCs) and Open Access Transmission Tariffs (OATTs) are both key components in the management and operation of electrical grids, but they serve different functions.

TCCs (Transmission Congestion Contracts): These are financial instruments that allow the holder to hedge against or benefit from variations in electricity prices between different locations on the grid. Essentially, they give the holder the right to "collect" or "pay" the difference in electricity prices between two points. TCCs help manage the financial risks associated with congestion on the electrical grid, which occurs when there's too much electricity trying to flow through a particular part of the system.

OATTs (Open Access Transmission Tariffs): These are sets of rules and rates that govern how transmission lines are accessed and used. They ensure that everyone has equal access to the electrical grid, regardless of who owns the transmission lines. OATTs lay out the terms, conditions, and rates for using the transmission system and are designed to promote fair competition among electricity providers.

Relationship: TCCs often operate within the framework established by the OATT. The OATT sets the ground rules for how the transmission system works, including how TCCs can be used or traded. In essence, the OATT provides the "playing field," while TCCs are one of the "games" that can be played on that field. Both aim to ensure that the transmission of electricity is efficient, reliable, and fair to all participants in the market.

Now I had better information. I frequently ask ChatGPT to enrich a prior response by combining more than one response into one.

"Combine the last two responses to provide a comprehensive explanation of how this relates to the following FERC agenda item EL22-64-000"

<Response is redundant to what is before and after...>

The answer now connects the concern with the response in terms I can understand but doesn't clarify the potential harm at stake. So, I ask the LLM to do that.

"Incorporate into the above description would NYISOs OATT might be unfair or unreasonable. Who is shouldering the risk and potentially handled unfairly?"

In July 2022, FERC initiated an investigation into NYISO's Open Access Transmission Tariffs (OATTs) to assess whether they were fair and reasonable. The concern was that the existing OATTs might lack adequate financial safeguards, particularly regarding Transmission Congestion Contracts (TCCs) and Financial Transmission Rights (FTRs). These are financial tools used to manage the cost and risk associated with transmitting electricity across different parts of the grid.

The question of fairness and reasonableness often revolves around who bears the financial risk. If the OATTs lack sufficient safeguards, the risk could disproportionately fall on electricity consumers or smaller market participants who might not have the financial capacity to absorb sudden price changes or defaults. This could make the system unjust, unreasonable, or discriminatory.

NYISO responded in October 2022, asserting that their existing rules and financial requirements are robust and effective. They argued that their system minimizes financial risk by requiring market participants to put up collateral. However, if FERC finds that these safeguards are not adequate, it could mean that NYISO's system is unfairly placing the burden of risk on parties who are not equipped to handle it, potentially leading to higher electricity prices or less market stability.

So, FERC's investigation and the subsequent agenda item E-3 aim to scrutinize whether NYISO's OATT is structured in a way that fairly allocates risk among all market participants. If it's found to be unfair or unreasonable, changes may be mandated to ensure a more equitable distribution of financial risk.

I could have dived into more detailed explanations of nuances like "volumetric minimum collateral". And I could do that in repeatable ways by continuing to refine my prompts.

That said, this answers my question in terms I can understand. I understand both the positions of the regulator and the responding ISO. I understand the equity at stake and the terms of art used by the participants. This took an hour - mostly - rummaging around FERC and NYISO's document archives for the first time. Now that I understand it, I could probably get to this in 15 minutes. The equivalent understanding would take hours without the AI assist.

If, I had to do this with any frequency, I would build out a Retrieval Augmented Generation (RAG) solution, using the orders and filings as the retrieved context and a Large Language Model (LLM) like OpenAI GPT or Llama 2 to generate the responses. For a small corpus of human readable text that is not privacy sensitive, I could do this pretty much turnkey. Privacy, scale and source document complexity would add the the effort and prohibit using the large public LLM's to improve the readability of the response. There are emerging toolkits like EyeLevel.ai that promise to ease and improve the parsing and chunking of the source documents and the relevance of the responses in those situations.