Gemini: look at my food and give me new ideas!

Cooking is a craft and a sensory experience, it's my hobby. Some weekends I choose to make something that I haven't made before and I enjoy it. But I worry that I default to similar techniques and cuisines: the fun is in the learning and trying new things! So, because I have been taking photos of my projects, could I ask? Gemini to evaluate the ingredients and regions the dishes come from, and then to suggest new things? Can I make it easier and quicker to share my projects on social media? Let's try!

From image to description

Take this Colombian Sancocho I made ( Per-Alexander L??k ).

I'll write code to get Gemini to explain what's there (I'll link to the repo at the end). And in fact I need to call this function on a whole folder of images - so when I call it, it looks like this:

python convert_images_to_descriptions.py --pre_prompt="here is a photo of a dish I made: " --folder_path="images" --post_prompt=" can you tell me what's in it? Describe each ingredient you see in the image and that will be needed for the recipe. Name 3 countries this dish may be from. In addition create 5 hashtags for this dish that reflect its visual aspect and cuisine"        

I ask for three things in my prompt: (1) the ingredients, (2) three countries the dish may be from and (3) hashtags for this dish.

So what does Gemini make of my dish?

This dish is a soup. The ingredients I see in the image are: - chicken - corn - potatoes - carrots - cilantro - avocado - salt and pepper. This dish may be from Colombia, Ecuador, or Peru. Five hashtags for this dish are: #Sancocho #SopaDePollo #Ajiaco #CaldoDeGallina #ComidaColombiana

Ok that's pretty good!?? Let's try another one:

This dish is a traditional Portuguese dish called Bacalhau à Gomes de Sá. It is made with codfish, potatoes, onions, garlic, parsley, and black olives. The potatoes are boiled and then mashed, and the onions and garlic are sautéed in olive oil. The codfish is then added to the onions and garlic, and it is cooked until it is flaked. The parsley and black olives are added at the end, and the dish is served hot. This dish is also popular in Brazil and Angola, which were all former Portuguese colonies. Here are five hashtags that reflect the visual aspect and cuisine of this dish: #bacalhauagomesdesa #codfish #potatoes #onions #garlic

That's impressive ????. One more: my actual meal last nigh

The dish is a Japanese clear soup with seaweed, tofu, and shrimp. The ingredients include: - bonito flakes - dried kelp - water - white miso paste - soft tofu - medium shrimp - green onion - fresh shiitake mushrooms. This dish is likely from Japan, Korea, or China. Here are 5 hashtags for this dish: #JapaneseCuisine #ClearSoup #SeaweedSoup #Tofu #Shrimp

I didn't use tofu, and the miso was red paste, but... I gotta give it to Gemini.

From description to JSON

Now if I want to organise all this, I need Gemini to go one step further. I need it to read its own descriptions and to put them in a json format! So here's how calling the function looks like:

python convert_descriptions_to_jsons.py --prompt="Take the following text and return a json file that has three keys 'ingredients', 'countries' and 'hashtags'. The values for those keys should be quoted from the text. Here is the text: " -folder_path="descriptions"        

And here's the ouput Gemini gave me for my dinner yesterday:

{
 "ingredients": [
  "bonito flakes",
  "dried kelp",
  "water",
  "white miso paste",
  "soft tofu",
  "medium shrimp",
  "green onion",
  "fresh shiitake mushrooms"
 ],
 "countries": [
  "Japan",
  "Korea",
  "China"
 ],
 "hashtags": [
  "#JapaneseCuisine",
  "#ClearSoup",
  "#SeaweedSoup",
  "#Tofu",
  "#Shrimp"
 ]
}        

Ok that's great! Now I have my photo turned into a json of ingridients, countries, and hashtags!

From multiple JSONS to my food habits

Now I can take the 70 photos of my own cooking experiments over the last few years and repeat this process: get Gemini to understand what's in the image and give me the ingredients, and countries of origin; then get Gemini to extract that information into a JSON format for each text description.

All that's left is to ingest all the JSON files and count the most frequent observations. So what are the countries I visit the most in my cooking experiments?

And the ingredients?

Italy, Spain, and Greece come from the frequent salads I make which are similar across the Mediteranean cuisine. And China and Japan from the soups and dumplings experiments. And I'm impressed Gemini can spot my frequent use of garlic, olive oil, and cilantro/coriader - even without seeing directly! ??

What about the hashtags?

From plots to new cooking ideas!

Now I can finally ask Gemini to help with new suggestions. Here's the prompt I passed:

I want to change my cooking habits and select something different from what I do now. This is a barplot of the countries you think my recipes come from <countries_barplot.jpg> And this is a barplot of the ingredients I most use <ingredients_barplot.jpg> Please suggest some dishes from other countries that are not represented in the barplot. And the dishes should avoid the ingredients here as well. Your suggestions should be healthy dishes, and be adventurous - avoid the most popular dishes and cuisines.         

So what did Gemini respond?

You could try making a traditional Ethiopian dish called injera. It is a sourdough flatbread that is made from teff flour and is typically served with a variety of stews and vegetables. Injera is a healthy and nutritious dish that is also very filling. Another option would be to try making a Korean dish called bibimbap. It is a rice dish that is typically served with a variety of vegetables, meat, and an egg. Bibimbap is a very flavorful and healthy dish that is also very easy to make. If you are looking for a more adventurous dish, you could try making a Peruvian dish called ceviche. It is a raw fish dish that is typically served with a variety of vegetables and spices. Ceviche is a very refreshing and flavorful dish that is also very healthy.

OK! I guess these are my next cooking experiments. ???? ???? ????

You can find the code here: https://github.com/graciofilipe/gemini-food