Automating proposals based on Historical Data and Whitepapers using Multi-Modal AI

By Houssame E. Hsain & Sayjel V. Patel

In this blog post, we’re taking a different approach. Rather than discussing our customer solutions, we’ll share how we’re addressing an internal challenge—creating proposals based on historical documents and whitepapers—and how this could evolve into a new use case for our enterprise clients.

Proposal Overload

Recently, as AI and enterprise software gain appreciation in the building industry, we've been overwhelmed by the growing number of proposals needed for clients, investors, mentors, and prospects. While this demand is a good problem to have, it requires a significant amount of work from our team, especially from the co-founders.?

Each proposal requires us to:

• Collate information from past projects
• Reference relevant information from our whitepapers
• Tailor content based on the specific Request for Proposal (RFP) or client emails
• Update content to reflect the latest insights and innovations

Over the past six months, we’ve found ourselves dedicating substantial time to the proposal process.

Can AI Help with This Problem?

As a software company increasingly immersed in vertical AI, we wondered if the techniques we use for our customers could streamline this task internally. Our initial research into similar solutions, particularly in finance and legal documentation, revealed limitations—existing tools focus solely on text-based information. However, our work requires integrating substantial visual information, such as diagrams alongside text, revealing an opportunity for innovation.

Drawing from these insights, we conceptualized a tool combining several advanced technologies:

• Multimodal Retrieval Augmented Generation (RAG)
• Vision Language Models (VLMs)
• Multi-agent AI systems

By integrating these tools, our goal is to productize knowledge from previous documents—such as RFP responses and whitepapers—and automatically collate relevant information into a comprehensive draft. While the tool isn’t intended to produce complete proposals autonomously, it provides a robust starting point, sparing us from beginning from scratch and combing through files for each proposal.

While the tool isn’t intended to produce complete proposals autonomously, it provides a robust starting point, sparing us from beginning from scratch and combing through files for each proposal.

Developing the Prototype

We kicked off the development of a prototype over a weekend with our team. At its core, the architecture comprises two primary components: a multimodal document indexing pipeline and a multi-agent response generation system.

• The indexing pipeline processes both textual and visual elements from PDF documents, converting them into rich embeddings via a specialized VLM and storing them in a multi-vector database. This enables advanced semantic matching, surpassing traditional keyword-based approaches by allowing the system to understand complex relationships between documents and queries.
• The response generation component employs a coordinated, multi-agent approach. It uses four specialized agents: an Executive Summary Agent for high-level overviews, a Technical Response Agent for detailed specifications, a Compliance Agent for requirement verification, and a Pricing Strategy Agent for cost structures. Each agent utilizes its own VLM for document analysis, maintaining context and focusing on its specific domain. The system also implements sophisticated context management, relevance scoring for document sections, and weighted document combination strategies to ensure coherent outputs.

Initial Results

This prototype has enabled us to generate tailored responses quickly, speeding up the document creation process and ensuring consistent, experience-driven communications. It demonstrated that we could leverage our past RFPs, whitepapers, and project documents to generate new, customized responses while aligning with DBF’s brand, design system, technical expertise, and innovative approach.

From a business perspective, the system significantly reduces RFP response times—from weeks to hours—while maintaining consistent quality across all sections. Its technical innovation, particularly in multimodal embeddings, allows it to interpret both text and visual content, like diagrams and charts. This enhanced comprehension of historical documents ensures accurate and contextually relevant response generation. The multi-agent architecture further enables specialized focus on each proposal section while preserving overall coherence, a notable advancement over traditional single-model document generation.

One standout feature of the system is its intelligent context management capabilities. These enable it to maintain context across multiple query iterations, ensuring consistency throughout the generated response.

One standout feature of the system is its intelligent context management capabilities. These enable it to maintain context across multiple query iterations, ensuring consistency throughout the generated response. This is achieved through relevance scoring, based on the Late Interaction technique from recent multimodal RAG research [1][2], and document combination strategies that preserve relationships between response sections while aligning with original RFP requirements. Ultimately, the tool accelerates the RFP response process while enhancing quality and consistency—a significant added value.

Next Steps - RFP Responder for Buildings

We believe this approach could also benefit our clients in the building industry. Visual communication and conceptual thinking are essential in building design and are central to our software environment. By connecting this with multimodal AI systems, we envision developing a solution to generate architectural documents like RFP responses and Basis of Design (BOD) reports by incorporating generative design and diffusion models.

Potential applications of this approach for our clients include:

• Automatically generating building layouts
• Creating diagrams and conceptual visuals that accompany written content
• Developing RFP responses tailored to specific requirements
• Enabling clients to produce BOD documents aligned with their guidelines

Our objective is to provide a solid foundation that can be easily customized, saving time and enhancing the quality of the output.

Conclusion

This internal project has opened new opportunities for vertical AI development for our clients. Automating repetitive parts of proposal generation frees up time for innovation and strategic thinking, empowering us to deliver fresh value propositions to our clients.

N O T E S?

1. Khattab, O., & Zaharia, M. (2020). "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT." https://arxiv.org/abs/2004.12832
2. ?Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., Hudelot, C., & Colombo, P. (2024). "ColPali: Efficient Document Retrieval with Vision Language Models." https://arxiv.org/abs/2407.01449