Accelerate Real-Time Solution Builds For the Enterprise with Airflow, Kafka, Docker, GitHub, TML, and ReadTheDocs

I have always been fascinated with things that move fast. Speed is exhilarating and gets us to places faster. Data and speed go hand in hand: Real-time data moves our world. This aspect of fast data that never stops flowing opens up tremendous opportunities for real-time data processing, machine learning and AI. However, many businesses around the world struggle with building real-time data solutions that are scalable, secure, easy to deploy, with proper documentation, with little to no code.

However, many businesses around the world struggle with building real-time data solutions that are scalable, secure, easy to deploy, with proper documentation, with little to no code.

The challenge with real-time data is that it flows and accumulates very fast, and this raises challenges for advanced processing, machine learning and AI. Not to mention that these real-time solutions can cost lot of money, consume lots of resources, and difficult to deploy.

What if, we could develop real-time solutions that can process real-time data with advanced machine learning and AI that are scalable, secure, well documented and can,

1. Automatically create online documentation for each solution - see here for sample documentation
2. Automatically commit all changes and scripts to GitHub
3. Automatically build and containerize each solution and push it Docker hub
4. Automatically build a streaming dashboard for your solution
5. Do it in less that 5 minutes

This is the TML solution studio or TSS. Here is a Youtube that explains the TSS: https://www.youtube.com/watch?v=z3h2nJXVgUs

Comprehensive documentation on the TSS can be found on ReadTheDocs: https://tml.readthedocs.io/en/latest/

The TML Solution Studio (TSS) With Airflow

The TSS is a revolutionary platform to build real-time solutions fast. It runs in a docker container, using 10 pre-written TML Dags that users can use to build real-time solutions in a matter of minutes. The TSS brings together IT/OT for real-time solutions. Source: https://tml.readthedocs.io/en/latest/docker.html

The diagram above shows how the TSS integrates with advanced technologies to create a tight process around real-time solution builds that automates many operational tasks like documentation, GitHub commits and logging, visualization, and containerization that enterprises struggle with. The TSS simplifies the entire end-end process of building real-time solutions for the Enterprise with 10 DAGS (Directed Acyclic Graphs) - these 10 Dags are shown in the Dag table.

The TSS simplifies the entire end-end process of building real-time solutions for the Enterprise with 10 DAGS (Directed Acyclic Graphs).

The TSS breaks down the building of real-time solutions to tasks that are defined in the 10 TML Dags as shown below. These Dags are pre-written, and all TML solution developers need to do is simply configure the parameters in these dags (source: https://tml.readthedocs.io/en/latest/tmlbuilds.html).

The diagram shows pre-written dags that:

1. setup the core parameters for your real-time solution
2. ingest data from any data source using: MQTT, REST, gRPC or local file
3. preprocess the real-time data
4. perform entity level machine learning (TML) and predictions on real-time data
5. create a streaming dashboard
6. deploy your solutions to docker container
7. integrate, in real-time, with AI
8. document your solution

The above process diagram shows connected dags in Airflow. These dags execute tasks that when executed together create your real-time TML solution in less than 5 minutes. Source: https://tml.readthedocs.io/en/latest/examples.html

The above process diagram shows connected dags in Airflow. These dags execute tasks that when executed together create your real-time TML solution in less than 5 minutes. TSS solution demo video can be viewed here: https://www.youtube.com/watch?v=ZzLL3tfBsh0

The TSS automatically commits all process scripts and code to GitHub, containerizes with Docker and creates documentation on Readthedocs.io for every TML solution. The TSS makes the building of large, enterprise ready, real-time solutions accessible to almost everyone.

The TSS dramatically simplifies the building of real-time solutions for enterprises, by automating core software development tasks and building a solid process driven approach to real-time solutions.

The TSS dramatically simplifies the building of real-time solutions for enterprises, by automating core software development tasks and building a solid process driven approach to real-time solutions that brings together IT and OT for real-time solutions.

Till next time..