The AI Chef App: Helping to Solve $1 Trillion Problem of Food Waste

Welcome to the 4th article in my genAI blog series. In the past few weeks, we've explored advanced prompting techniques for genAI. Now, it's time to see these models in action.

Tackling Global Food Waste

Food waste is a pressing global issue with far-reaching consequences:

• Environmental Impact: Food waste accounts for 8-10% of global greenhouse gas emissions. Annually, about 1.3 billion tons of food are wasted.
• Economic Costs: The financial toll of food waste is staggering, with a global cost estimated at $940 billion per year.
• Social Implications: The food wasted in Europe and North America alone could feed the world's hungry population four times over.

In light of these facts, I aimed to create a simple AI application for daily use as I have caught myself throwing away food as well.

GenAI to the Rescue

The solution involves a simple method to capture home ingredients, which are then inputted into a Large Language Model (LLM) to generate recipe suggestions. This process starts by taking a photo of the open fridge with a mobile, followed by uploading the image to the app. Recognizing that not all ingredients might be visible and that no system is flawless, the app allows for the manual addition of items not detected or those stored outside the fridge. The flexibility to modify the prompt is crucial, enabling users to specify their desired cuisine type or accommodate dietary restrictions - these are important in my household.

Chef AI App in action:

Battling food waste while supporting creativity

This approach not only offers a practical solution to minimize food waste but also inspires creativity in the kitchen, turning the challenge of using up ingredients into an opportunity for culinary exploration.

High level App architecture:

First, I would like to mention that the app demonstrates usage of several different models: GPT 4 Vision (Image -> Text), GPT 3.5 Turbo (Text gen.) as well Dalle (Image gen.). Each of them play a unique role while holistically creating a personalised journey for the user to provide accurate and personalised recipes.

1. Flask Framework: Utilizes Flask for handling web requests, routing, and rendering dynamic HTML content based on user interactions, such as submitting ingredients or image URLs.
2. OpenAI API Integration: Leverages OpenAI's GPT models (GPT4 Vision + GPT 3.5) for analysing image from your fridge + recipe generation and DALL-E for image creation, directly integrating these capabilities into the app's workflow to enrich user experience.
3. Dynamic Content and Configuration: Employs Jinja2 for dynamic content rendering in response to user inputs and environment variables for secure and flexible app configuration, including API key management.

Step 1: Get image of your fridge

The first step is to capture image of your fridge and upload it in the app so automated AI classification of ingredients can be performed by GPT 4 Vision model.

Step 2: Upload the image of your fridge

Upon form submission, a Flask route handling POST requests captures the image URL from the form data using Flask's request.form object. This URL is then passed to a function that constructs a request to the OpenAI API, specifically targeting a model capable of interpreting images, to extract and list visible ingredients from the provided image. Multi-modal GPT 4 Vision is used for this purpose.

Step 3: Ingredients recognised + human input

Once the image analysis is complete and ingredients are identified by the OpenAI model, the application renders a new HTML page displaying the list of detected ingredients using dynamic content rendering with Jinja2 templating. The user is prompted to review and modify list of ingredients through an interactive form, ensuring accuracy of the input for recipe generation. This step includes user feedback into the process, allowing for the addition of ingredients that might be outside of the fridge or were not captured correctly. Also, the user is asked about their favorite cuisine, allergies and number of recipes to generate in order to provide personalised experience.

def get_ingredients_from_image(image_url):
    response = client.chat.completions.create(
        model="gpt-4-vision-preview",  # Ensure this model is correct and accessible
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "List the ingredients visible in this image as a comma-separated list. Example: chicken, milk, potatoes"},
                    {
                        "type": "image_url",
                        "image_url": image_url,
                    },
                ],
            }
        ],
        max_tokens=300,
    )
    return response.choices[0].message.content        

Step 4: Recipe generation incl. images

Leveraging the refined list of ingredients provided by the user, the application constructs a tailored prompt for the OpenAI GPT model (3.5 Turbo is sufficient vs. GPT 4) , requesting the generation of recipes based on ingredients captures. Upon receiving the textual recipes from the GPT model, the application iterates over each recipe, calling the OpenAI DALL-E model to generate images that visually represent the dishes. These recipes and their associated images are then dynamically integrated into the final HTML page using Jinja2 templates.

prompt = f"Show me {no_recipes} healthy recipes for a dish with only the following ingredients: {ingredients}. Choose recipes only from {cuisine} cuisine. Per recipe, list all the ingredients used, no {allergies}."
    completion = client.chat.completions.create(
        messages=[
            {
                "role": "user",
                "content": prompt,
            }
        ],
        model="gpt-3.5-turbo",
    )

    response = completion.choices[0].message.content        

def generate_recipe_image(description):
        try:
            image_response = client.images.generate(
                model="dall-e-3",
                prompt=description,
                size="1024x1024",
                quality="standard",
                n=1,  # Request only one image
            )
            return image_response.data[0].url
        except Exception as e:
            print(f"Error generating image: {e}")
            return None  # Return None or a placeholder image URL in case of an error

    # Generate an image for each recipe and collect the URLs
    recipe_images_urls = [generate_recipe_image("Visualize this dish: " + recipe) for recipe in individual_recipes]

    # Pair each recipe with its corresponding image URL
    recipe_data = list(zip(individual_recipes, recipe_images_urls))

    # Pass the paired recipe data to your template
    return render_template('results.html', recipe_data=recipe_data)        

This content draws inspiration from existing materials and practices. As an employee of Microsoft, I want to clarify that the views and interpretations presented here are my own and do not necessarily represent the official policies or positions of Microsoft. This is intended for educational and informational purposes only.