Can ChatGPT do my old job?

Alternate title - "ChatGPT as a universal interoperability layer"

I spent thousands of hours analyzing legal documents in order to reduce them to high-value, structured, data. Some of the tasks involved parsing hundreds of pages of orders, but a lot of them were very straight-forward text processing tasks.

Some engineers I know have used tools like CoPilot and ChatGPT as a "virtual pair programmer," and I got to thinking, what about my data extracting tasks? I decided to try something very simple, which is creating structured challenge claim data from PTAB filings. For the non-Patent folks, these are administrative challenges before the USPTO to invalidate some or all of a patent's claims. The most common type of action is an IPR, which seeks to invalidate claims under §102 or 103 of the Patent Code.

Since this entire administrative court system was created by patent attorneys, it is rigorously structured in terms of process and filings, much like patents themselves.

I made this task very simple by just grabbing a few Petitions from the USPTO, then feeding the Challenge Statements into ChatGPT. I've read there are ways to formulate prompts based on long documents, but it's kinda fiddly and I didn't have time to figure it out. I realize this makes the task substantially easier, but I wasn't looking to test document parsing. 

Overall, this should be a relatively easy task because PTAB challenges, and patent claim structure are already numerically coded.

After about 14 tries (I am *not good* at prompt writing), I was able to get chatgpt to generate the JSON I wanted. Over the previous iterations, ChatGPT would muddle the output structure, leave out keys (it never wanted to include the claims array in the same json), and I had to wipe the chat a few times and just start over because once ChatGPT started down a bad path, it was hard to navigate out of it.

ChatGPT did some strange things with number sequences vs expressing spans with a dash. It was inconsistent between runs, but overall, this looks plenty structured to me.

Now, I wanted to see if ChatGPT could apply the same prompt to another challenge statement, so I tried a second Petition. This one had an added wrinkle, in that the statutory grounds are not explicitly called out in this text - "Anticipation" is a §102 ground and "Obviousness" is under §103. An IPR Petition is always a challenge under these two grounds, but the specific grounds here only use the phrases "anticipated by" and "obvious over", not including the specific patent sections in it's arguments. How would ChatGPT handle this?

Well, I wouldn't still be typing if it wasn't interesting! After a bit of confirmatory praise (who doesn't appreciate gratitude?), I asked ChatGPT to repeat the process with this challenge:

As you can see, "Grounds for Unpatentability" includes the correct statute, even though the prompt does not. Delightful! Also, the claim numbering has changed again. Mysterious!

Is it from context derived from the first prompt? General patent knowledge baked into its transformers, able to map anticipation and obviousness? Maybe it's just guessing? I don't really know, so I pulled down a third Petition to try again.

Alas, the third petition contained table data, which doesn't encode into a readable form in the PDF. I couldn't even copy-paste and get all of it. No way to ask this of ChatGPT without re-writing it on my own, at which point, I'm pretty certain it would get the right answer. I know there are separate ML tools for text extraction from tables, but I don't have access to one.