2024 - My Year in Review

I'm introducing my first-annual Year in Review! This is to help friends and colleagues learn about what I've been working on and investing effort in. 2024 has been an extraordinary year both at work and in my personal life. As I'm writing this, I'm overwhelmed by a sense of gratitude for my family, friends, and colleagues who were with me along the way. I don't think there's a single accomplishment which doesn't owe more to someone else than to myself, so calling it "my" Year in Review is, in that sense, misleading. However, this is not generically "the" Year in Review, nor does it report on anyone else's experience of these things, so we'll have to content ourselves with that pronoun. I've broken this down into Professional Highlights and Personal Highlights.

Professional Highlights

My work as a field consultant at Ab Initio this year has been all about enabling a smarter, faster, more robust, more intuitive, more scalable, and more sovereign data platform at existing and prospective customers.

1. Advancing the Vision for "AI Everywhere"

What an exciting year it was for artificial intelligence at Ab Initio, both in terms of making our products smarter and enabling businesses to develop AI-driven decisioning with our software.

My first contribution was to revisit the text-to-SQL work from my time at Yale, dust off the Spider large-scale cross-domain dataset we had hand-annotated five years ago, and enable the benchmark to work in the context of our Data Catalog, as the rest of our talented team did the heavy-lifting to build out a RAG pipeline for text-to-SQL (-to-Easy>Data Graph). By standing tall on the rich business metadata from our catalog, and producing Easy>Data graphs and Guided Expressions rather than just textual SQL, the user receives an executable artifact which is visual, intuitive, can be further developed and tested. This enables extremely ergonomic natural language interaction with data. I also helped prototype air-gapped configurations, implementing the RESDSQL approach as a GPU-enabled standalone RESTful service, which runs on-premise.

Second, I developed a lecture on GenAI entitled, "The Bit-Bang: How Information Became the Center of the Universe", which debuted on Center Stage at Schwarz IT's DataCon in Sofia, and then was revamped and again featured at our own Customer Connect Frankfurt for our German customers. The talk was not simply a technical overview of neural networks, transformers, and retrieval augmented generation, but I also presented both the broader historical context in terms of information theory, and philosophical outlooks such as the ability for models to achieve true understanding and consciousness rather than "simply" emulating intelligence.

Third, the time was ripe to really enable our customers to build AI into their decision processes. To do that, we needed intuitive examples of how to perform retrieval augmented generation with Ab Initio Graphs. By collecting much prior-art from my outstanding fellow GenAI practitioners, I curated the "GenAI Examples" repository, which exercises vector embeddings and search, prompt engineering for a large language model, and real-time decisioning with monitoring. I unveiled this as a "cooking show" at Customer Connect Frankfurt, where we went from raw ingredients to a production-ready application within a 15 minute live demo. Then I used the same repository to prepare an internal off-site training at the Berlin Field Meeting.

Finally, during my visit to the Lexington headquarters early this month, I presented for an internal audience of 300 on the topic of AI enablement at our customers - what we've done, which projects are underway, and what the unique value-add of doing AI on our platform is. I expect 2025 to be the year of moving from idea and proof-of-concept to fully-integrated, end-to-end solutions.

2. Achieving Data Sovereignty with Open Table Format

Our customers need to remove their dependence on on-premise and Cloud SaaS databases, which both own your data and significantly monopolize the compute workloads by allowing SQL as the only way to get data in or out. Europe is also broadly pushing IT sovereignty by developing their own cloud offerings like STACKIT, and independent storage is a critical puzzle piece.

Open Table Format, best implemented by the Apache Iceberg community, gives you the illusion of having a database - complete with table definitions, ACID properties, schema evolution, and time-travel - despite simply using open-source file formats on cheap and reliable cloud object storage. Most importantly, it gives you full ownership of your data.

The project we've set out on is a Streaming Data Lakehouse built on Iceberg, where we will be ingesting everything ranging from transactions data from JMS to logistics/delivery data from SAP systems into a unified landing zone in Iceberg - with updates happening in "near real-time" (1 min - 30 min depending on the need) in order to drive faster business insights. From there, downstream data pipelines will further enrich the value of these raw sources.

We presented the initial architecture and experiments at Customer Connect Frankfurt, have demonstrated several key proofs-of-technology since then, and plan to go live early 2025.

3. Enabling Business Groups with a Robust Data Platform and Intuitive Marketplace Experience

The logical next question after democratizing your data with Open Table Format and the Ab Initio Streaming Lakehouse is, how are business users empowered to use that data to drive insights and AI decisioning? In 2023, we built out the Self-Service Business Intelligence (SSBI) Platform at Schwarz, which involves cataloging all the enterprise data sources, enabling trustworthiness through lineage and the Knowledge Graph (e.g. technical terms linked to business terms), and ultimately accessing the data in business and AI-friendly tools like SQL, Python, Tableau, and our own Easy>Data Graphs. We presented about this project at the international meeting in Bath, England.

In 2024, the rollout of SSBI kept sailing smoothly, and we conquered a few remaining challenges like SSO-based identity federation to follow cloud-native security practices, and importing new metadata to the catalog on dramatically faster timeframes, while also enabling custom sources like folder-file partitioned datasets. But one major user story remained, which was how to truly scale enterprise-wide with an intuitive data discovery workflow, which we call the Data Marketplace.

A Data Marketplace is built on top of the Data Catalog while involving two new workflows with metadata: Data Products and Data Contracts. A Data Product is an arbitrary group of datasets or schemas, complete with automated provisioning of data access, monitoring of data quality and the data's origins, and an optional linkage to a Data Contract. A Data Contract specifies the terms of use for the data, such as the business division, a retention period, encryption at rest, and so-on. This architecture and set of workflows allows IT teams to focus on platform stability and new features while enabling the business to assess and access data in a more agile, self-service, and scalable way, without sacrificing on security. We have settled on a vision and achieved a POC this fall, and 2025 will be a big year for rolling out the Data Marketplace.

4. "On the Road Again": Sales and Events

In addition to project work, I had the privilege to represent Ab Initio in a sales capacity at several conferences, including DataCon Sofia, TDWI Munich, BDWI Kongress Berlin, Data Festival Munich, and CloudX Summit Heilbronn, and I was on-site for in-depth demo sessions and discussions with several prospects. I also staffed four Ab Initio Certification events in Munich, helping customers and partners become officially certified practitioners with our software.

Personal Highlights

1. Literature and Religion

My proudest achievement is completing a two year long close-reading and weekly discussions of the Hebrew Bible. My father and I set out on this journey together early last year, working from the remarkable literary translation from Robert Alter, available here. I also deepened my fanboy-hood of C.S. Lewis, reading A Grief Observed, The Problem of Pain, Reflections on the Psalms, Surprised by Joy, and Miracles. My return to faith and religion is not something I can explain here, but it has enriched me beyond words, and I would love to talk with you individually.

I also continued my study of Spanish Literature with the remarkable Prof. Jose Manuel Magraner Herreros. We read and discussed, among others, Niebla by Miguel de Unamuno, Crónica de una muerte anunciada by Gabriel García Márquez, and La Biblioteca de Babel by Jorge Luis Borges.

2. Mathematics

Mathematics has long been a passion project and hobby of mine, but this was a particularly awesome year in math. I published a public GitHub repository Coding Exercises Inspired by Enderton's Mathematical Introduction to Logic, which includes two Jupyter Notebooks to explore some of the key results of sentential and first order logic. I also conquered about 80 (or 90 if you allow hints) of the delicious math puzzles in The Price of Cake: And 99 Other Classic Mathematical Riddles, and I haven't given up on the remaining 10 top-difficulty riddles. Finally, as the perfect harmony between love of coding and love of math, I (re-)discovered Project Euler and made a small dent in this amazing collection of problems.

3. Sports and Fitness

Many of you know I suffered a knee injury a few years ago, and although I didn't need knee surgery, I was on crutches for months and had a prolonged cognitive block preventing me from really trusting the knee. After the "real" pain was gone, I felt ongoing phantom pain. At the low point, I was sure that I would never ski or run again, and I just hoped I would relearn how to walk without that faint, barely-noticeable limp, and the awful expectation of pain from the simplest of motions.

Due to the patient, never-failing encouragement from my family and girlfriend, consultations with leading orthopedic surgeons in Munich, and diligent physical therapy, my knee is now better than ever. I was skiing and running many times this year, biking to work each day, swimming, hiking up mountains - having all the adventures and more which I wished to have in my twenties.

4. Travel

Speaking of adventures, this year was an epic year of travel! It started with three weeks in South Africa, including doing a safari in Kruger, visiting Cape Town, and driving the Garden Route. I also had an amazing five year college reunion, came home for Thanksgiving, and visited the Lexington headquarters twice. I added Bath, Prague, Sofia, Zagreb, Zadar, and Trogir to the list of remarkable historic European cities/towns I've experienced.

5. Miscellany

I attained permanent residency in Germany, got my German driver's license, published my personal website https://daveyproctor.com/, and celebrated 5 years at Ab initio.

A Note of Thanks

As I said, none of this would have happened without my family, friends, and colleagues. Therefore, please accept my most heart-felt thanks and deepest gratitude. I wish you a season filled with joy and reflection, a very merry Christmas, and a healthy, happy, and prosperous start to 2025. There's nothing we can't do, together.