5 Tips for Creating a Data-Driven Architecture

Introduction

This article provides an overview of a technique that has proven useful over many years (yet most projects/people rarely use it). This article is not an exhaustive description of each and every detail required to implement a data-driven architecture; doing so would be very lengthy. Besides, one size does not fit all, so any attempt to provide that level of detail here would be far too presumptuous. My hope is that this article triggers some ideas on your end to help you design and build more resilient systems that are easier to maintain and serve your customers better.

What is a "Data-Driven Architecture?"

A data-driven architecture (DDA) is a solution whereby the information/values displayed in a system and the system behavior are determined by metadata. And this metadata is easily updated to implement changes dynamically within the system without rebuilding the application and deploying code. Only database changes are needed to change displays and/or behavior at a moment's notice. The idea is to decouple business logic from code and use data to drive the system. However, code provides the construct for using the data to manage the user interface/user experience (UI/UX) and system behavior. Think of code as the container and data as the contents in the container.

Why would I want to build a "Data-Driven Architecture?"

We've all worked on systems that have evolving requirements and shifting priorities, so anything we can do to accommodate those changes makes life easier for us and better for our clients. We want simple updates to be simple and our clients want seemingly simple changes to be quick. DDA does just that … it allows us, as developers, to rapidly implement predictable, or semi-predictable, changes using a proven infrastructure.

What kind of things can be controlled with a "Data-Driven Architecture?"

Okay, at this point you know what DDA is and why DDA is valuable. But what kind of things can you control with this approach? Short answer is anything you want! Longer answer is anything you want that doesn't impact performance too much and doesn't over-complicate things unnecessarily. Here are some examples to give you an idea of where data can drive processes:

• Front-end webpages to display field labels, show/hide values, and populate UI controls such as drop-down list boxes (including the sort order for values), check boxes, radio buttons, etc.
• Reports and dashboards to populate filter controls such as drop-down list boxes.
• Extraction, Transformation, and Load (ETL) routines to implement business rules, exclude/include specific records, and connect data points through crosswalks.
• Environment configurations to allow the same exact code to run in multiple places (i.e., Development, Test, Production) … write once, deploy anywhere.

Helpful Tips

Tip #1: Identify the system attributes you want to control dynamically.

Proper requirements analysis should reveal potential areas that could benefit from dynamic behavior. These areas are generally characterized by statements such as:

• "Most of the time these values apply but they can, and do, change."
• "That's going to change with new legislation."
• "Ugh, we have to change that record/setting/file every time we deploy the system to a different environment!"
• "Those values never change." ??

You get the idea, things that seem volatile either because they are fluid or contract/expand based on business circumstances (e.g., fiscal year rules, dollar thresholds, pre-defined lists of values, role-based access controls, etc.).

Tip #2: Create reference tables to store metadata that drives system behavior.

Create database tables to store code-value pairs where the code is the primary key and the value(s) are the description(s). Create values for short descriptions and long descriptions because some screens/reports may use the short description, such as for drop-down list boxes, whereas other screens/reports may use the long description to provide more information. Also include a sort order column to order the values in the front-end because alphabetical is not always the desired order. For example, you may have a list of values that includes 'Other' and you want 'Other' at the end of the list. If any values begin with 'P' or beyond, then 'Other' will be nestled in the middle of the list. Adding ORDER BY sort_order to your SQL SELECT statement retrieves the list of possible values in the order desired. Here's a sample reference table followed by sample reference data.

For environment configurations, you can create a reference table to store environment-specific settings. For example, maybe different SMTP servers are used in each environment for sending e-mail messages within the system or different URLs are used for references to SharePoint sites/Google Docs. All of this can be easily implemented in the DDA to avoid hard-coding in release builds for simple changes. You should NEVER modify code for the purposes of getting the system to work in another environment! Remember, your goal is to write once, deploy anywhere.

Tip #3: Develop abstraction layer to interface with domain data.

Create database views to access domain data (i.e., business data, not reference data) that allows you to control how the data is joined, converted, and returned as a ready-to-use result set. Direct table access is acceptable but always consider the option of using views. Views can protect the front-end (screens & reports) from innocent changes and allow business logic to change in one place … the view. For business logic that determines if something is true or not, use database functions. And for business logic that needs to perform a series of steps, set values to arguments passed by reference, and/or return result(s), use stored procedures.

Database views, functions, and stored procedures all provide a level of abstraction that makes it possible to implement changes in the database with little to no impact to the front-end code. Deployment of these changes are much simpler and quicker because there are no builds or releases to prepare since they are simple database updates. And that doesn't mean we are abandoning or circumventing change management processes and source code control, it just means that the entire process finishes quicker with greater confidence.

Tip #4: Develop front-end, and any APIs, to use the abstraction layer and reference tables.

All front-end code, whether it's screens/webpages, reports, or dashboards, should use the abstraction layer. And more importantly, the abstraction layer should also be used for system-to-system interfaces. Two or more independent systems requiring integration should use this abstraction layer to streamline exchanges and mitigate unintended impacts to each other. The abstraction layer not only prevents the need for each system to [unnecessarily] understand the inner-workings of the other but also protects each system from potentially harmful changes. The published Application Programming Interfaces (APIs) for each system safely control the entry/exit and what can/can't be done between systems.

Tip #5: Perform purposeful test-and-fix cycles.

Make intentional changes to the reference data that will make it clear if the front-end is using DDA or not. Areas that do not pass testing are areas that need to be reviewed, fixed, and re-tested. You should plan on several iterations of testing to make sure DDA is properly implemented throughout the system. Here are a few examples to test the implementation:

• Create really long descriptions in the reference data. In the sample ref_employee_type table above, you can temporarily change the 'FT' emp_type_short_desc to '*FULL-TIME EMPLOYEE CODE*'. Then any screen/webpage, report, etc. showing the short description should show '*FULL-TIME EMPLOYEE CODE*'. If it still shows 'Full-time', then you know that code is not using DDA and must be fixed.
• Create unusual values to make it clearer that values are bogus for easy identification. For example, temporarily change the short description above to something like '0123456789012345678901234'. Also be sure to use the maximum length available to not only test that the code uses DDA, but also to test how the longest possible values appear.
• Verify processes or workflows still work as intended to make sure developers used the primary keys to drive system behavior and no other reference table attributes. For example, a process/workflow for Full-time employees should check for 'FT', not the short description, long description, or sort order. Changing any of those non-key attributes should not affect any processes/workflows; if it does, then that code-base needs to be fixed.

This testing will pay many dividends later because once DDA is implemented and working, future metadata changes won't have to be tested. A quick check of metadata updates can be easily done and then deployed, which is why it's critical to do thorough testing before-hand. Skimping on testing will cause more work, and re-work, later, so do proper testing! Doing so will reduce your sustainment efforts, accelerate delivery of changes, and please your clients.

Summary

These five tips should be enough to get you started on your journey of designing and building systems driven by data. The little bit of extra thinking and designing up front will save lots of time in the future. Once the construct is built and tested, it's a matter of simple database updates.

In closing, here are the key benefits of using DDA:

• No code changes for routine updates.
• No downtime because you're updating data, not deploying code.
• Quicker turn-around times because you're simply adding/updating/removing data.
• Little to no testing because you're re-using the infrastructure that's already been built and tested. Typically, it's just a quick look and you're done.
• Quickly adapt to changing requirements because updating data is much faster than writing/testing code.
• Happier clients/users because you can respond to their requirements at the speed of business.

I hope that's enough to get you started on your journey to building a robust system driven by data more than code. Feel free to drop a comment or send me a message if you have any questions.