Our Picks of High-Performance LLM Tech Stacks

Stacks we choose to build and deploy Large Language Model applications

Over the past few months, We've successfully built and deployed various applications utilizing large language models (LLMs). Here are a few examples:

• Banking assistant: AI-powered chatbot to handle customer inquiries and support requests for financial institutions.
• Medical clinic onboarding system: Conversational AI to guide new patients through registration and initial information gathering.
• Legal document analyzer: Automated tool that reviews and extracts key information from legal texts and contracts.
• Kindle library natural language search: Application that allows users to find books and content in their e-reader library using conversational queries.
• AI-assisted code editor: Intelligent tool for quickly navigating, modifying, and refactoring large codebases in software development projects.

Two of these applications are currently in production, serving real users.

Through extensive trial and error, I've honed a tech stack that I find to be the most effective for building these types of applications.

The Ultimate LLM Stack

TypeScript

TypeScript forms the backbone of my stack. Its type-safe environment significantly enhances developer experience and maintainability, allowing seamless type definitions across both frontend and backend. This was a game changer compared to my early projects that used Python for the backend and a different framework for the frontend.

Instructor

Instructor?ensures typed LLM responses, providing predictability and consistency in outputs, crucial for robust application development.

LlamaIndexTS

LlamaIndexTS?excels in document processing and retrieval, making it a powerful tool for extracting data from structured documents and generating embeddings.

Milvus

Milvus?offers efficient vector storage, making it ideal for managing vector data. I recommend starting with Milvus Lite during development and scaling up with Zilliz for production environments.

MongoDB

MongoDB?is my go-to for a flexible, scalable database solution. Its native support for JavaScript objects aligns perfectly with TypeScript, streamlining data management.

Next.js

Next.js?serves as my full-stack framework, providing server-side rendering capabilities and excellent performance out of the box. Its stable API allows for seamless integration with Copilot or GPTs for coding assistance.

Stytch

Stytch?is my choice for authentication, offering robust B2B-focused features like OAuth and Magic Links, crucial for enterprise applications.

Logging+Eval

For LLM logging and analysis, a service like?Velvet?is essential. While I haven't used Velvet personally, its positive feedback suggests it’s a reliable choice for storing and analyzing LLM API responses.

Vercel

Vercel?is my preferred hosting platform, thanks to its seamless integration with Next.js and excellent performance monitoring tools. It simplifies deployment and scaling, allowing me to focus on development.

HappyDevKit

HappyDevKit?handles feature flag management efficiently, enabling controlled rollouts and A/B testing of new AI features, which is crucial for iterative development.

Sentry

Sentry?provides robust error monitoring and alerting, ensuring application reliability by sending alerts directly to Slack and email.

Key Takeaways: Recommended Tech Stack for LLM Systems

• Language:?TypeScript
• LLM Library:?Instructor
• RAG Tool:?LlamaIndexTS
• Vector Storage:?Milvus
• Database:?MongoDB
• Full-stack Framework:?Next.js
• Authentication:?Stytch
• LLM Logging: Logging+Eval (TBD)
• Hosting:?Vercel
• Feature Flags:?HappyDevKit
• Error Monitoring:?Sentry

These tools and frameworks have been instrumental in my development of B2B applications leveraging LLMs. While this stack works well for me, your needs may vary. Experiment with different combinations to find what works best for your projects. I'd love to hear your thoughts and experiences!