Bridging the Gap: Engineering Solutions in the Age of Over-Engineering

In today's tech landscape, over-engineering is often the default. Engineers, captivated by the allure of the latest and greatest, sometimes rush to implement solutions without first assessing their actual necessity. I've been there, drawn by the shimmer of new tech, only to realize that the YAGNI (You Aren't Gonna Need It) principle is a mantra I should've embraced earlier in my career.

While developing the Housemates Connect API, I was tasked with creating a webhook solution. This would allow API users to subscribe to specific events and receive data, particularly in the context of contract changes.

To give you a snapshot of our infrastructure: we operate what can be termed a "modern monolith." This structure, underpinned by a shared multi-tenant database, has its roots in legacy systems. While it presents challenges, it also offers benefits, particularly in infrastructure simplicity. This allows us to focus on delivering impactful solutions to the student accommodation sector without being bogged down by infrastructure concerns.

However, I envisioned Housemates Connect as a bridge—a hybrid between our core system's monolithic architecture and the microservices' agility. This standalone application serves as an intermediary layer between our API users and our core system. Fetching data from the core and relaying it to our API users was straightforward. The challenge arose when we had to write back to our core system, especially for tasks like processing enquiries and bookings from API users.

Given the tight integration of our CMS and other third-party tools with our core system through custom integrations, Housemates Connect couldn't operate in complete isolation. Our workaround? We maintained unique IDs that referenced data in our core database. By leveraging queues, we could cross-update data using these IDs, ensuring consistency and integrity.

For the webhook implementation, while sharing queues between applications seemed like an initial solution, the complexity of our queue operations made it unfeasible. Instead, I opted for HTTP calls to transfer relevant data between the two systems, establishing clear demarcations between them.

This approach has proven effective. Not only does it maintain the integrity and efficiency of our operations, but it also offers real-time feedback to our API users. They can now pinpoint issues directly from their dashboard, understanding both the nature and cause of any problem.

In conclusion, while the allure of the "new" in tech is undeniable, it's essential to ground our engineering decisions in practicality and purpose. The Housemates Connect API journey reinforced this belief, reminding me that sometimes, simplicity and clarity trump complexity.