PDF Generation Service - Efficient PDF Generation from DOCX Templates using LibreOffice - A Fast Alternative to Puppeteer with Few Limitations.

We are super lucky to work in an environment that allows for first-time thinking and idea generation and this week our very own Scottish-based Engineering Lead Francesco Belvedere will share the following solution that was developed to address a real-life challenge.

The topic in question:?

PDF Generation Service - Efficient PDF Generation from DOCX Templates using LibreOffice - A Fast Alternative to Puppeteer with Few Limitations...?

As this topic has not been covered properly anywhere on the internet or other sources, Engineering then had to produce a unique solution. We are all about empowering the collective knowledge base within Fintech and think this will be of great interest to Devs and Solution Architects who follow the Nomo Fintech page feel free to share and tag us in this distribution of this article.?

This article gives a real insight into how challenging the build is and the opportunity our Engineering teams have in doing the ability to do cutting-edge solution generation. Have a detailed read below:

Generating PDF documents flexibly is a hard problem to “get right”. A popular approach is to use Puppeteer, a Node.js library that allows controlling a browser instance from the command line, however, this comes with some major drawbacks. A less common approach is to use LibreOffice, an open-source technology that can quickly generate PDFs with some key advantages. In this article, we explore the solution some teams at Nomo are using to generate PDFs and we discuss the pros and cons.?

Goals

When thinking about our next document generation service, these were the goals we wanted to accomplish:

• Be able to use dynamic headers and footers for our documents e.g. to contain personalised branding.
• Use some form of string interpolation to hydrate the document by passing custom data to it.
• Validate the data that is used to hydrate the document, to ensure API clients conform to the data schema.
• Update the document or its layout without coding skills.?
• Avoid changing the code and re-deploying our backend for every edit or correction to the document copy.
• Keep the generation fast.

The Problem with Puppeteer and HTML/XML-based PDF Generation

Puppeteer's PDF generation functionality relies on loading a webpage in a headless browser and “printing” it as a PDF. This approach can be slow and resource-intensive since a full web browser must be started (although the browser instance can be long-lived) and a DOM loaded before being able to “print” what we see in a PDF format. This gets particularly intensive for complex documents with many elements and images.

Solutions to convert from HTML or XML to PDF without starting a browser like wkhtmltopdf, pdfkit, dom2pdf and more, also come with their own challenges since dealing with pages, margins and overflow can become a nightmare to work with and maintain.

The Solution: LibreOffice to the Rescue!

An alternative approach to PDF generation is to use LibreOffice (LO), an open-source technology that offers a word processor, spreadsheet, and presentation software, as well as tooling and a command-line interface to convert documents to different formats, including DOCX to PDF.

DOCX is a file format used to store documents. It is an XML-based file format and can contain text, images, tables, and other types of content.

Using DOCX docs allows:

• Easy visual editing of templates. HTML document updates require the right skill set, whereas edits to DOCX documents, be that for text, formatting or spacing, are often easier to carry out for non-developers and allow for a shorter feedback cycle.?
• Fast processing times. Converting DOCX templates with LibreOffice is quicker for larger files.

Following, let’s explore how LibreOffice fits in the bigger picture of a document generation service.

Document Generation Service

Step 1: Create a DOCX Template and an Optional Layout

You can use any word processor to create DOCX layouts and templates, such as Microsoft Word or LibreOffice Writer. The template should include placeholders for the data that will be replaced during the PDF generation process (this is also known as string interpolation). For example, you could include placeholders for the name, address, and date in the format {{property_name}}.

Step 2: Install LibreOffice

Install LO locally using your package manager or download it from the official website. For your live system, you can spin up a LO service or use a ready-made solution like https://github.com/unoconv/unoserver or https://github.com/gotenberg/gotenberg, which provides a REST API on top of LO.?

At Nomo, we run the document generation in AWS Lambda and we interface with LibreOffice (LO) binaries that are hosted in a Lambda Layer via the built-in LO CLI. To keep the generation fast and prevent AWS from wiping the temporary storage where LibreOffice is unpacked, we use provisioned concurrency and ping our lambda every 5 minutes. However, going forward we plan to separate the LibreOffice install and try out one of the wrapper solutions mentioned in the paragraph above.

Step 3: Validate the passed parameters using JSON Schema?

Alongside our template and layout files, we host a JSON Schema file that describes which properties are required to build the document and what their type or format is. This schema is used at runtime to validate the data passed to the generation service.

Step 4: Combine and Generate the Complete DOCX Document

A DOCX document is essentially a zip or collection of XML files that are bundled together. Each part of the document is usually represented as a separate .xml file and has the following structure:

.

├── _rels

│ ? ├── document.xml.rels

│ ? ├── footer2.xml.rels

│ ? ├── header1.xml.rels

│ ? └── header2.xml.rels

├── …

├── document.xml

├── footer1.xml

├── footer2.xml

├── header1.xml

├── header2.xml

├── media

│ ? └── image1.png

├── styles.xml

├── theme

?????└── theme1.xml

For maximum flexibility and reusability, we use layouts and templates which we can dynamically combine such that the resulting file replaces:

• header#.xml, footer#.xml - these contain custom header and footer header, and are taken from the layout document.
• document.xml - this is the body of the specific document, e.g. Bank Confirmation Letter doc body, and is taken from the template document.

To obtain the final combined document, we follow these steps:

1. Use jszip to deserialize the layout and template DOCX files and get back strings containing XML.
2. Load the strings with xmldom to easily manipulate the XML.
3. Insert the template (body) XML into the layout XML via xmldom.
4. Serialiase the combined XML back into a string using xmldom.
5. Use handlebars to replace the predefined placeholders (e.g. {{name}}) with the dynamically passed data.
6. Use jszip to bundle back into a DOCX document the original XML files with the modified XML layout and template.

At this point, we have a fully usable and formatted DOCX document. You could stop here if all you needed is a DOCX document with a custom header and footer, and placeholders replaced by values.

Step 5: Convert the DOCX Document to PDF

Finally, since we have a working DOCX document, we can now use LO to convert the file into a PDF. If you are using LO locally, you can do this using the LO command-line interface. For example, this command will convert the template to a PDF and save it as document.pdf:

soffice --headless --convert-to pdf document.docx

Results

We created a flow that addresses some of the original goals we had in mind. Work is still needed to achieve some of the other goals, and this approach comes with a few caveats too:

• Use dynamic headers and footers: this was possible by combining separate DOCX documents at runtime.
• Custom input data validation: using JSON Schema is straightforward, although extra care is needed to keep the versions of JSON schema and the DOCX document aligned.
• Document hydration with custom data: easy to achieve with handlebars. However, when it comes to formatting the data (think datetime formatting) the result could be confusing as rules are contained in the DOCX doc, too.
• Update the document copy or layout without coding skills. This is possible but also comes with gotchas:?
• Some text processors (e.g. Word) like to insert extra tags and line breaks as you type, meaning that some placeholders might not work properly! Some settings might need to be disabled depending on the text processor used.?
• Margins and tables can be tricky to sort out when e.g. looping through records of data.
• Avoid code changes and re-deployments: this is possible by hosting the DOCX and JSON schema files externally such as on AWS S3. However, extra care is necessary to deal with versioning. And because these are bundled documents instead of plain text, it is more difficult to compare docs across versions.
• Keep the generation fast. We are hosting the logic above in AWS lambda with roughly 3GB size and achieving an average 1.35s total completion time for lambda invocation + layout and template combination + DOCX to PDF conversion. We expect this to be faster if we did not need to combine layout and template DOCX documents.

In conclusion, we feel this solution offers good performance and is a good compromise to achieve some of the goals that we set ourselves, although like with every other approach to PDF generation, this comes with its own set of challenges.

By Francesco Belvedere