Putting the “Visual” in Data Visualization: Using Generative AI to make complex topics relatable

Problem

According to Gartner, only 44% of data and analytics teams are effective in generating value for their organization. Too often, analytics teams get stuck in a cycle of high-effort, low-impact reporting work that drains time, energy, and budget. The problem of a business having too much data, but not enough insight, is similar to motivating people to take action on climate change. According to the Yale Program in Climate Change, we’re “drowning in information but starved for knowledge” due to an overwhelming amount of information that is difficult for individuals to apply to their daily lives. As a thought experiment for today, I would like to consider an alternative approach.

Opportunity

What If: there was a way to use generative AI to take complex information and make it compelling?

Scenes From Tomorrow is an experimental proof-of-concept web app built for that purpose, to tackle climate climate procrastination. The app is available here, the code is here, and a few general use cases for this pipeline are outlined below.

Hypothesis

The target audience is people concerned about climate change but unsure how to take action in their daily lives. The hypothesis is that by providing information that is contextual to the user’s location, impactful (by being multi-modal), and actionable, the app can empower procrastinators to take action. More updates to come in the future as that hypothesis gets tested.

Architecture & Data Flow

The core data flow is outlined below. We want to retrieve data for the user based on a location, explain what it means through text, and then show a visual.?

Technical Requirements?

Here is the stack this app was built on:

• HuggingFace Spaces: for managed hosting including GPU if needed
• Chainlit: An LLM web app framework integrated with LiteralAI for data layer support. Optionally, LiteralAI's data hosting can be exported for analysis such as a relational database on AWS.
• Lightning.AI: a versatile development platform for working quickly and deploying things in the cloud. With Lightning.AI, I was able to create the app, build the docker image, and then push to the HuggingFace Spaces repo to deploy the app.
• Probable Futures API: the data contributor to this project. Their API includes an extensive set of weather indicators for each location queried and warming scenario
• ChatGPT through OpenAI: LLM for the assistant and adaptation Q&A
• Stable Diffusion: A text-to-image model invoked through a HuggingFace endpoint; using the StableDiffusion XL Turbo checkpoint to get high-quality completions in ~2-3 seconds (with GPU acceleration).?

?

Set Up

Difficulty: Medium and up

This process assumes you are familiar with LLM development, including assistants and function calling, as well as docker deployments. A few requisites before starting this:

• Build an assistant: Matthew Harris provided an excellent repo late last year that walks through the basics of setting up an assistant and connecting it to Chainlit. That article is a good starting point although both Chainlit and OpenAI APIs have had updates since then.
• Account set-up: Don’t forget the core steps like this getting an OpenAPI key and loading it into your .env. On HuggingFace spaces this as abstracted out as a ‘Secret’ connected to your account but if running locally or on Lightning.AI you’ll need the typical .env method. Also make sure to add your Literal API key to the .env file to capture session analytics using Literal’s built-in data layer!?

User Session Data Flow

Let’s take a step-by-step walkthrough of how the process works. Our hypothesis has 3 elements: provide contextual insights, in an impactful multimodal format, with clear and actionable next steps.?

Part 1: Make it Contextual

• Getting the Data: First, the user will request the information for a specific location through the chat. From a UX perspective, we assume the user knows the purpose of this app before they arrive.?
• Turning it into a narrative: Here’s where the storytelling kicks in. In this app, the agent has been provided with instructions on to summarize the information, guided by the excellent Probable Futures handbook, which helps individuals understand climate change through its impact on heat, water, and land. I put in some additional guidance and prompt examples to deliver the information with brevity.?

Part 2: Make it Multimodal?

• From Words to Images: The next step in our process is to create an image that aligns with the data provided. The two steps to do this are text-to-image model, and prompt.
• The text-to-image model we use here is Stable Diffusion XL Turbo, which allows reasonable high-quality image completion in around 2-3 seconds, with GPU acceleration.?
• To get an appealing and relevant image from SDXL, there is a further prompt refinement step which creates a prompt for the image model based on the location and its weather data, the narrative provided to the user, and best practices for prompting SDXL Turbo.
• These elements get wrapped in a function that generates and presents an image to the user. ?

Showing impacts over time with the Image-to-Image method

• “Scenes From Tomorrow” gets its name from the concept of showing people a progression of scenarios: today, in which the climate is already 1.5°C warmer vs. last century; the 2.0°C warming future which looks likely, and the 3.0°C potential scenario we hope to avoid.
• To create these scenes, I used the image-to-image method of Stable Diffusion. This takes the image created in the first prompt (the 1.5°C scenario), and then refines it through a new prompt for the 2C and 3C images.?
• The new prompt comes from the data returned by the PF API for each scenario. The app identifies the biggest change within the new dataset and turns that into a prompt for the image-to-image model. For example, a region that will experience a high increase in rain and flood risk will get a message to that effect; while an area that will see an increase in the number of hot dry days as its most prevalent feature will get a prompt described an area impacted by drought.

Part 3: Make it Actionable

• Interactive Q&A and Actionable Takeaways: Finally, we want to give the user a clear understanding of how climate change will impact their location and lifestyle.?
• Using Chainlit’s built-in functionality, the app includes some QA handling where users can ask follow-up questions. This personalizes the experience to their specific needs.

• The session ends with a personalized summary of what the user can do to adapt, prepare, and mitigate the impact of climate change on their location and lifestyle. This summary is created by the LLM using a combination of the information returned from the Probable Futures API and the questions asked during the session.?

Finally, a few thoughts on generalizing this more broadly through additional use cases:

Customer Analytics and Segmentation

• Those in the field of marketing analytics will appreciate this use case. Analytics teams spend a lot of effort to segment their customers and then humanize those segments with illustrative stock photos and iconography. Synthetic visuals could allow nuanced and impactful ways to communicate segments within the company and help internal audiences take action.

Sales Analysis?

• Imagine a summary report and accompanying BI dashboard describing key trends in product sales. Visualizing product sales through the context of customer use cases and shopping journey moments could bring an impactful and compelling moment in a meeting at risk of running long and dry (like a quarterly review).

Image Guardrails?

• One final note. You may be asking about the quality and accuracy of images for use in a business context, along with questions of brand safety. Stay tuned for my next blog post which will share a Jupyter notebook showing how to use a vision model to do evals on generated images and refine the PromptEng and generative model hyperparameter pipeline to ensure safe content generation.

Please try out the app, share feedback and also leave your thoughts in the comments below. This app can be an intense experience but will hopefully motivate audiences to stop procrastinating and take action on climate.