Experiments in summarization using LLMs

For many months I have been trying to be on the top of the AI news, it is damn hard ! So much is happening at such fast pace, it feels like before I can make sense of what happened last month, 10x more things happen in the current week. As a VC, I need to be on the top of the game - I need to be informed of what is happening with respect to startup funding, product launches, new reasearch papers, SOTA models, new paradigm shifts e.g. SFT, RFT, Reasoning etc - whole alphabet soup. And more interestingly more interesting uses of AI that people are talking about. Or the more intellectual AGI / job losses / LLMs for India / AI sovereignty debates etc.

At last count I have subscribed to over 60+ newsletters, I read some 100+ blogs, twitter handles, youtube channels etc. It has become information overload - just reading takes away so much time, that getting time to process knowledge is hard. (Imagine reading even one paper of arXiv takes up at least an hour if not more)

So I needed a way out, and I felt the most efficient way is to code my way out of it. (I have been programming since 5th grade - BASIC games on a PC-AT in the 90s). So I came up with the idea of scraping all these 100+ websites, 60+ newsletters, etc.. Before OpenAI Computer Use and Anthropic CUA came out in the last quarter, I was coding Playwright scripts with GPT3.5 - and I can tell you it was hard. So I had to simplify, so I hit upon the idea of just subscribing to lot of RSS feeds (wow a two decade old technology coming in handy) - but there was another problem some of the interesting new stuff is not available on RSS or most publications force you to come to the main website by just sending you to a small snippet on their RSS feeds.

So finally three weeks ago, inspiration stuck !

I realized that a lot of interesting newsletters / substack / medium etc are anyways coming into my inbox. Why not just parse them and collect insights from them. I can tell you now there is a human acceptable that is working now. But I have stepped into the minefield of 100K is not enough context window, LLM evals, prompt engineering, APIs vs Ollama, GPT vs Reasoning models - who knew markdown handling is also not easy.

I plan to opensource the project. But the most interesting architecture debate I ended up with is this choice - which is the cost vs quality vs coding speed trade off between these three choices.

I can tell you that the first approach is the current MVP - it works ok for all newletters I receive in 24 hours - currently they amount to around 70K tokens (~55,000 words across 12 newsletters on average). But the approach goes for a toss if you want to do a weekly review - context length issues. Or even within a daily review, some of the interesting things get missed - I have seen using o1 vs 4o is dramatic improvement. (But of course how much are you willing to spend for your daily newsletter, which is just ok quality 10 cents or 50 cents?)

Have tried approach two as well, but it is not working well at all. The challenge is that newsletter contents are varied and the golden prompts are unable to capture that heterogenity - so the output is a bland summary. The newsletters contain funding rounds, research papers, product launches, news etc. I also tried an OpenAI suggested recipe in their cookbook - but it wasn't good at all, actually quite bad. (I was told the recipe was meant for the turbo models era !)

So finally I'm currently working on the tedious approach of creating custom prompts for each different newsletter - and man that is hard - it involves, multiple iterations with GPT (again o1 is better than 4o) and a bunch of wrangling with Cursor (I want to reduce tokens by doing a bit of parsing of html newsletters).

If you have any suggestions feel free to comment.