Attribute Extraction from Images using DSPy

Introduction

DSPy recently added support for VLMs in beta. A quick thread on attributes extraction from images using DSPy. For this example, we will see how to extract useful attributes from screenshots of websites

Signature

Define the signature. Notice the dspy.Image input field.

Program

Next define a simple program using the ChainOfThought optimizer and the Signature from the previous step

Final Code

Finally, write a function to read the image and extract the attributes by calling the program from the previous step.

Observability

That's it! If you need observability for your development, just add langtrace.init() to get deeper insights from the traces.

Source Code

You can find the full source code for this example here.

Langtrace x DSPy

Langtrace natively supports the tracing and monitoring of key metrics from DSPy optimizers and pipelines. This is helps you with understanding how a chosen module or an optimizer from DSPy works under the hood and gives you key visibility into better optimizing the performance of your application.

For more information, check out our previous blog post on this integration here. Here are some additional threads that people have found helpful:

• Curated list of examples built using DSPy.
• Grokking MIPROv2
• Building Compound AI Systems using DSPy and CrewAI.
• Instruction tuning for a Rule based chatbot using DSPy.
• Building a Reliable Summarization system using DSPy.

Useful Resources

• Getting started with Langtrace https://docs.langtrace.ai/introduction
• Langtrace Github https://github.com/Scale3-Labs/langtrace
• Langtrace Website https://langtrace.ai/
• Langtrace Discord https://discord.langtrace.ai/
• Langtrace Twitter(X) https://x.com/langtrace_ai
• Langtrace Linkedin https://www.dhirubhai.net/company/langtrace/about/