How We Built ChartAI in 7 Days

?? ChartAI is a free online tool, that allows you to upload a screenshot of a chart and then to easily customise it. ChartAI was built by Milan Wittpohl & Sarah Behrens as a fun side project in one week, and is available via chart-ai.app.

1. From Pain to Idea

A few weeks ago, we were working on a report for our studies at Tomorrow University of Applied Sciences . We needed to include various charts from external sources, each requiring slight adjustments—like translating labels to English or changing colours. These tasks were not only time-consuming but also frustrating, triggering that part of our brains that asks,

"Why is there no tool for that?"

This experience sparked our curiosity and motivated us to explore the possibility of creating a solution that could automate these tedious processes. We were hooked on the idea of developing a tool that could simplify chart customisation and make our lives—and the lives of others—much easier. However, as our current focus is on identifying opportunities to start a climate tech startup, we decided to only spend up to a week on this.

2. Market Research: Are There Existing Solutions?

The challenge we set out to solve was clear: How can we make an image of a chart editable when you don't have the original data? To tackle this, we embarked on some classic Google research, asking ourselves if such tools already existed and how they might function.

Our search led us to various resources, including tool websites, blog posts and YouTube videos, which described intriguing approaches to data extraction from images. We also delved into Google Scholar to find academic insights into this area. What we discovered was a range of tools with semi-automatic capabilities. These tools allowed users to upload an image but often required manual calibration of axes or the selection of data points. Some solutions were limited to working with vector images only. Additionally, we found that all the tools we encountered were quite complicated to use and lacked a fun, engaging interface.

While these tools showed that our idea was feasible, they also indicated a gap in the market for a fully automated solution. This initial research provided us with the confidence to move forward, knowing that there was potential interest and demand for a tool that is able to recreate charts based on images.

3. Feasibility Check: Can AI Recreate Charts?

At this point we started to shift focus towards further evaluating the feasibility. Building a solution for recreating charts based on images falls into the category of products where

1. there is not only one way to accomplish the task
2. you can quickly find yourself in a very complex software architecture
3. it might simply not be feasible, at least not in a one week timeframe

Pragmatic Feasibility Testing

In order not to over-engineer anything, we tried to start simple, using existing tools for the feasibility check without writing a single line of code or setting up a project.

Can We Use AI to Significantly Accelerate the Development?

We wanted to use AI for this project as a way to “outsource” most of the heavy lifting. Only if AI could do most of the work, it would be possible to accomplish this project within one week.

We started by simply uploading screenshots from charts to ChatGPT and Google Gemini and asking the AI to extract the data. The results where very mixed and okay at best, but it showed that current LLMs are generally capable of extracting data from charts.

Can We Offer an AI-Tool for Free?

At this point we paused the technical feasibility for a moment and focused on the financials. It was likely that ChartAI had to be for free to one attract users and two because the product would probably not be good enough to charge money for it. We also knew that due to the time constraints we would not consider training or hosting our own model. This means that ChartAI would rely on the API of OpenAI or Google. However, the state of the art models GPT-4o and Gemini 1.5 Pro would become expensive quickly. Luckily for us, there is a current trend in the AI world of creating highly capable but significantly smaller and cheaper models (e.g. GPT-4o mini or Gemini 1.5 Flash). After a quick cost modelling with a spreadsheet we knew that such models were financially feasible. Back to testing the models.

The mixed results from earlier indicated that it would require far more time than one week to build a truly great product. But this does not mean a good product is not possible. We shortly considered fine tuning an existing model but creating the training data alone would take too much time. Therefore, we tried to improve the results with better prompting.

The AI of ChartAI

At this time we considered what the desired output of the AI should be. We wanted the user to be able to upload an image of an existing chart and then be able to download a customised version of the chart. One could think that you could simply ask the AI to generate an image based on the screenshot and the customisations of the user. However, image creation with AI is

1. significantly slower and more expensive than text generation
2. not very precise, which is very bad for our use case
3. not good when it comes to rendering text, also very bad for our product idea

If ChartAI is not creating an image of a chart, it needs to output something that can be converted into a chart. In the past we had worked with ChartJS which is a library for creating all sorts of charts. There are definitely other libraries and also other approaches out there, however, it is our believe that if you want to build something quickly, you should use the tools you are most familiar with. People tend to use new and shiny tools just because they are hyped. While it is definitely important to stay up to date, in projects with a short timespan you want to move as efficiently as possible. Use whatever you feel comfortable with. For us that meant using ChartJS (alongside NuxtJS, typescript, tailwindcss and vercel).

Having said that, at this point we still haven’t written a single line of code. We knew that programming the UI is definitely feasible, the AI part was the tricky one. We asked the AIs to extract the data and then to write ChartJS code.

The results were catastrophic. The AI either failed to extract the data correctly or created broken ChartJS code. Awesome.

At this point we were close to cancelling the project, but we decided to put in a few more hours and to further improve our prompting. In the end the AI was able to create good results most of the time. We accomplished that by creating a multi-step prompting process:

1. In the first step we ask the AI to extract the data from the image and then returning the data in a specific JSON format. This format “helped” the AI to extract the relevant information. You can think of it like a template that the AI fills, “helping” it to work more structured.
2. In the second step we give the AI the context of the first step (without the input image for cost reduction) and then ask it to generate ChartJS code. Again, we provided the AI with a basic JSON format to write the code. However, the AI often used old ChartJS syntax or simply produced broken code. As ChartJS is open source we analysed the code and identified specific files (TS typings) that would be the most helpful for an AI to learn the tool. We then added these files to the context of the AI. The results were significantly better. We later added some logic within ChartAI to further improve the robustness of this step by fixing the most common mistakes the AI made.

In the end we had a pragmatic solutions that worked well enough for this MVP. We ended up using Gemini Flash as we got slightly better results compared to GPT-4o mini. This process taught us the importance of relevant context for an AI and how giving AI models structure can significantly improve quality.

4. Product Scoping and Feature Design

In designing ChartAI, our goal was to create a user-friendly interface that maximised usability while minimising complexity. We opted for a chat/text interface to empower users with maximal flexibility in customising their charts without relying on traditional buttons and tools for adjusting colours, fonts, and other elements.

To ensure a good user experience, we knew we needed an engaging way to show users what was possible, especially since there were no specific functions for making adjustments to the chart. We identified four common use cases for modifying static charts: translating text, changing the look and feel, adapting data points, and transforming the chart type.

We integrated an example within the app to illustrate these adjustments and added a button that provided personalised AI-powered tips based on the user's specific chart image. This approach simplified the user experience while fostering creativity and engagement with ChartAI.

5. Desirability Check

Discovering the problem ourselves and finding solutions online were just the first indicators of desirability for ChartAI. To validate this further, we reached out to friends and fellow students to see if they faced similar challenges when writing reports or doing presentations for their studies or work. Their feedback confirmed our suspicions: we were onto something valuable.

With our backgrounds in product development and user research, identifying and analysing potential user needs has been a core focus. We believe in the importance of thorough user research, employing different methods during various project phases. However, given the low implementation effort required, we decided to build ChartAI as a beta tool. This approach allows us to learn from real users, observing how they use the tool under realistic circumstances and optimise the tool based on their feedback.

6. Implementation: Content Creation and Development

After confirming the feasibility and refining our product concept, it was time to bring ChartAI to life. At this stage, we divided our tasks to leverage our strengths effectively. Milan took charge of the technical implementation, ensuring the tool's functionality and efficiency, while Sarah focused on design tweaking and content creation.

Despite being a small project, we believe that attention to detail is crucial for crafting unique user experiences and creating those delightful "wow" moments. This project also provided an excellent opportunity to experiment with new tools and develop skills that might be valuable in the future. For instance, we created a 3D animation for the loading visualisation using Spline and produced a product video with Tella.

This collaborative effort allowed us to build a tool that not only meets user needs but also offers an engaging and enjoyable experience. By combining technical expertise with creative design, we aimed to make ChartAI both functional and fun to use.

7. Marketing and Communication

While building ChartAI, we were already strategising on how to introduce our tool to potential users. Our first step in marketing and communication was to define our target groups and create tailored content formats and topics for different platforms to reach them effectively. We drew a lot of inspiration from Pieter Levels' marketing approaches, which we highly recommend exploring for innovative ideas. (Link in the comments).

To kick off our marketing efforts, we posted an introduction on LinkedIn, leveraging our existing network in the digital sector. We also decided to share the development journey of ChartAI on this platform, engaging our current followers who are primarily professionals in the digital space. Additionally, we launched ChartAI on Product Hunt and Hacker News, where we received some valuable feedback and responses.

By utilising these platforms and strategies, we aim to connect with a broader audience and gather insights that will help us refine ChartAI further. We're excited to see where this journey takes us and how ChartAI can evolve with the input of its users.

How We Were Able to Built ChartAI in 7 Days

Looking back we believe that the following decisions enabled us to built ChartAI in just one week:

1. We asked ourself every single day very critically, if ChartAI was worth pursuing. This helped to set realistic goals and stay on track.
2. We always tried to take the most pragmatic approach, like testing the feasibility as much as possible without writing code.
3. We aimed to balance being practical while also putting effort into small details like design or animations.
4. We leveraged existing strength and experiences as much as possible, like Milans knowledge in software architecture and engineering as well as Sarahs strong focus on market research and product design.
5. We collaborated very closely together and shared insights continuously along the way, e.g. showing each other how competitors products work or which prompts generate good results.

About us

We are Milan Wittpohl and Sarah Behrens , passionate for climate impact and digital innovation. With our strong backgrounds in building digital products as a software engineer and product manager, we are driven by a shared desire to create real, lasting impact through digital solutions that address the pressing challenges of the climate crisis.

Building ChartAI has been an exciting and enjoyable side project for us, serving as a creative break from our primary focus on in-depth research into climate impact topics like smart grids and energy management.

ChartAI allowed us to explore our passion for innovation while developing a practical solution to a common problem. It also provided a fantastic opportunity to share our experiences and insights with others.