DATA ENGINEERING/DATA ARCHITECTURE

DATA ENGINEERING/DATA ARCHITECTURE

By W H Inmon

In recent vintage there has been much discussion about data engineering. Most of the discussion has centered around the pioneering work done by Joe Reis and Matt Housley. Joe and Matt recently released a new book on data engineering that focuses on the issues and techniques of data engineering. It is an excellent guide and it is highly recommended.

Today’s focus on data engineering is an outgrowth of the earlier focus on data science. For some unknown reason, the data scientist – at least most of them – tried to do analytical processing in a vacuum. The data scientist insisted on working alone. The data scientist acted as if no one else in the corporation knew anything about data. It is like a surgeon trying to do operations without the aid of the anesthesiologist. The surgeon knows that the anesthesiologist is necessary to keep the patient alive during the operation. Without the anesthesiologist the operation may be successful but the patient may be dead. The data scientist thought that he/she could work in a vacuum without the help of anyone else. As such the data scientist turned into a glorified data administrator or a data garbageman. The data scientist spent the vast majority of his/her time wrangling with data and not doing data science.

In truth, the great promise of data science has abated as business has learned that data science has a hard time delivering on its promises. In the meantime the data scientist grapples with the problems of –

??Producing tangible and significant business valued results

??The inadequate analytical tools used by data science, such as NLP

Some organizations still support the myth (and have the hope) that data science is going to lead to great results. But they are waiting for the kind of results that justify the high cost of the data scientist. They have been waiting a long time and it has been an expensive wait. ?They are still waiting.

Today the focus is on data engineering and that change in focus is a very welcome change.

But as the world focusses on data engineering it is wise to remember that data engineering has an equal ?partner in its journey. And that partner is data architecture.

In truth, data engineering and data architecture are like the classical yen and yang. Data engineering

needs data architecture and data architecture needs data engineering.

In order to understand this symbiotic relationship, consider this – neither data engineering nor data architecture is fruitful without the other. A data engineer that doesn’t understand architecture is lost in the desert. In order to be a successful engineer, the engineer must understand the roadmap that has?

been laid out by architecture.

And vice versa, the data architect who makes beautiful renditions is worthless unless those renditions can actually be implemented. The data architect needs the data engineer in order to make the data architecture come to life.

In order to paint the wall, both the paint and the paintbrush are necessary.

In the rush to embrace data engineering it is easy to forget that data architecture is an equal partner to data engineering.

Bill Inmon lives with his wife and his two Scotty dogs – Jeb and Lena -in Denver, Colorado. Jeb is royalty. He is from the Agnelli kennel. He lets his little sister know that he is royalty all the time. She is a great Scotty but she is a commoner.