Generating Images with Large Language Model (GILL)

By now, we all know that LLMs work by creating embeddings of sentences in a large, multi-dimensional textual space.

While LLMs like GPT4 are able to take in multiple types of data including textual, images, audio and video, their outputs are text-only.

When we look at text-to-image generation models, they similarly have their own text to image embeddings.

Is there a way to bridge this gap?

Seems there IS a way to make two different embeddings talk to each other. This is akin to creating convertors between say the Google and Apple universes. Impossible you would say? It isn't so in this particular instance.

A recent paper titled "Generating Images with Multimodal Language Models" shows the way. So how do they do it?

They use pre-trained LLM as well as the image generator models to produce both image and text outputs. A large majority of the parameters are kept constant with only a small number of parameters on image caption data to be fine-tuned. There are three main challenges that they overcame

• The model needs to learn to process image-and-text content
• It needs to learn to produce images (retrieved or generated)
• It needs to determine whether to produce text or images at each step.
• Finally, it needs to decide whether to generate an image or to retrieve an image through similarity search

1. This is done by taking in image and associated text inputs and converts them to text interleaved with image embeddings as shown in the figure on the left. The important piece to note here is that this mapping translates images into embedding vectors in the token embedding space of the LLM that is being used.
2. In the next stage, the Model output embeddings are passed through a decision model that decides whether to return an image from the database using a similarity search or whether to use a GILLMapper Model to generate a new image.
3. This last piece is done by mapping the IMG tokens from the original embedding space to a semantically meaningful region of the input space of an image generation model like Stable Diffusion.

This process ensures that using pre-defined LLM and image-generation models a mapping can be defined, allowing "conversation" to happen between the two models with different embeddings. Isn't that something!