Lessons Learned from Transforming an Open-Source Project into a SaaS Solution Using Large Language Models

I recently participated in an initiative to take an open-source saas project from GitHub and transform it into a fully-fledged commercial application, complete with subscription and billing functionality. This journey provided fascinating insights into the practical use of LLMs for professional development, particularly when comparing Claude AI’s Sonnet 3.5 and ChatGPT o1. Although this anecdotal comparison is far from a rigorous scientific study, it highlights both the strengths and pitfalls of these models in real-life coding scenarios.

The Project at a Glance

Our end-to-end solution involved:

1. Selecting an Open-Source Project: We chose a promising platform on GitHub that had solid core functionality but required significant re-architecting for multi-tenant SaaS and subscription features.
2. Defining the Architecture: We designed a scalable solution with robust user management, subscription billing, and automated deployment pipelines.
3. Extending the Platform: We integrated new capabilities to handle tiered subscriptions, usage metrics, and dynamic feature toggles across user groups.

Throughout the process, we relied on two LLMs—Claude AI’s Sonnet 3.5 and ChatGPT o1—to generate code, optimize workflows, and help troubleshoot issues. This dual-LLM approach allowed us to compare their performance in near-identical tasks, with the caveat that each model comes with its own unique training and usage constraints.

Keeping the Models Focused and “In the Loop”

One of the biggest challenges we faced was keeping the models consistently aware of the most critical files for the problem at hand. When you’re dealing with large codebases and multiple functionalities, LLMs can get “confused” or produce incomplete answers if they lose context.

Our Key Learnings

1. Focus Each Chat Session We discovered it’s far easier to write code with LLMs when each chat session is focused on a specific, self-contained task—such as “create a new API endpoint” or “write a test suite for the subscription module.” We found that writing code and testing it within the same session helped maintain consistency and context.
2. Establish Ground Rules at the Start Each new conversation would begin with a prompt that re-established the “rules” for our collaboration. We asked the LLM to:

• Always refer to the project knowledge base for definitions, best practices, and file structure.
• Always provide the full path of the file(s) to be updated or created.
• Always ask critical clarifying questions if any part of the prompt is ambiguous.

This significantly reduced the amount of guesswork on the LLM’s part and streamlined the coding process.

3. Maintain a Centralized Knowledge Base We kept a knowledge base that included:

• The full file tree of the project.
• The business objectives and requirements.
• A condensed version of the open-source platform’s documentation.

We repeatedly reminded the LLM to consult this knowledge base, because if we didn’t specifically instruct it to do so, the model would sometimes ignore or overlook the content. This step greatly helped in producing consistent and contextually correct code.

Anecdotal Comparison: Sonnet 3.5 vs. ChatGPT o1

Though this is not a formal or scientific benchmark, we noticed the following patterns:

1. Code Accuracy
2. Context Management
3. Prompt Sensitivity Both models were quite sensitive to prompt quality. Detailed prompts that included the rules for code style, references to the project knowledge base, and clarifying questions yielded significantly better results.

Conclusion

In this real-world project, Claude AI’s Sonnet 3.5 stood out for its ability to produce fully executable and error-free code around 60–70% of the time, especially when dealing with self-contained tasks. It did show signs of strain when the required functionality was spread across multiple files, which demanded repeated context-setting and careful instructions.

Nevertheless, both LLMs proved invaluable in accelerating development. By breaking tasks into discrete chunks, keeping a shared knowledge base, and prompting the models with very clear expectations and rules, we effectively harnessed the power of large language models to transform an open-source project into a SaaS solution with subscription features.

While not a formal study, these anecdotes underscore the promise of LLMs in a professional development environment—and the importance of structured prompts, thorough documentation, and a well-maintained context to keep these models on track.

Have you tried using LLMs in a similar way? I’d love to hear your experiences and any tips or tricks you’ve discovered for maintaining context and consistency in a complex codebase. Feel free to share your thoughts in the comments!