Micro Service in Data Ecosystem

Background:

Data Micro Services are key part of modern Data Ecosystem. In general, there are multiple API types we may support.

·?????? Search APIs

·?????? Business Application Integration

·?????? Self Service APIs

·?????? Knowledge Graphs

There are typically 4 layers of any enterprise Data Eco-system. (There can always be some variations of this arrangement based on Industry, Line of Business etc.)

Data:

It includes the bottommost but the important component. It will include all your Enterprise Canonical data, Data Governance and Data Quality components around it. This data can be in structured, semi structured, or unstructured format. The sourcing can be batch based, near real time or real time basis. Data need not be sourced. only from Internal Source Systems but can also come from Niche Market Data providers. Last but not the least Data Layer will have due consideration for any historical data as well.

Foundation:

This layer will be characterized by Data Ecosystem and Applications. This is the layer where API strategy should materialize. This strategy will enable the foundation and support connecting/building capabilities.

This layer will be further powered by Data Management Dashboards, Entitlements and Security.

This layer is characterized by 3 types of data:

1.????? Bring your own data / Sandbox style.

2.????? Fit for consumption / Ad-hoc data / Market Industry Data

3.????? Canonical / System outputs

This layer will provide obvious values.

·?????? Data Cataloging

·?????? Accessibility via interactive Query engines like Athena

·?????? Advance DQ Control

·?????? Documentation Definitions

·?????? Discoverability of data

·?????? Schema on Demand

?

?

·?????? Enterprise Data APIs

?

Capabilities:

1.????? Should support for:

2.????? Data Science Tools,

3.????? Analyst Support,

4.????? Discovery,

5.????? Robust IT platform

6.????? Applications and Dashboards

7.????? Data Discovery Tools (ex:Alation..)

8.????? Adhoc Dashboarding and Analytics (Ex: PowerBI , Tableau, MS Excel ..)

9.????? Data Science ( Jupyter, Sower, TensorFlow , Python …)

?

Portal:

Portal is where capability integration is fueled with

1.????? Discoverability

2.????? Flexible Access

3.????? Single Sign On

4.????? Advanced Data Widgets

…..

?

?

Microservices a key underlying component for building the Data Ecosystem

?

In a typical large-scale enterprise this is how use cases are segregated:

?

Target Use Causes (80%+)

??????? Batch

??????? Change Data Capture

??????? Near Realtime

??????? Bulk Histories

??????? Analytical Data Sets

Glue, Raw Files,Parquet Files, Athena

Data Science, Search

?

Target Use Causes (5%+)

??????? Read intense.

??????? Rapid Queries

??????? Complex Reporting/Cube

??????? Curated expanding on Glue.

?

?

Redshift, Snowflake, Materialized Data Marts

Advance Analytics Tools , Legacy Systems

?

?

Target Use Causes (15%+)

??????? Real-time

??????? SPA/Applications

??????? Queries

??????? API Wrappers

??????? Writing Information (Create, Update, Delete) – Business Logic/Workflow

Kubernetes / Docker / MicroServices

Use Cases: Extracts, Cloud Native App..

?

Key Microservice Components

Foundation should be packed with re-usable components, from Infrastructure, application patterns, to code/libraries

Key Stack Components:

n? Restful microservices hosted on Containers.

n? Kubernetes to Orchestrate the containers.

n? service mesh for the microservices

n? Full local Development with Docker/Linux

n? Build Test Deploy Jenkins (CI/CD) pipeline for everything.

n? Re-usable basic AUTH/Z component

?

n? Building common accelerator “sidecars” for logging, auth, metrics, monitoring, etc

?

?

Guiding Principles for MicroServices

–????? Reduce data hops

–????? Microservices

n? Is Independent and encapsulated.

n? Fulfils a specific business objective.

n? Run anywhere (containerized and mesh based)

n? Maintainable by DevOps/Two Pizza teams

–????? Event Driven - Avoid batch mind-set instead (Close of exchange vs end of day)

–????? Simple Common Patterns

n? Sequence

n? Snapshot

n? Search (Lookup)

–????? Define service type layers.

n? Business/Process

n? Data/Entity

n? Utility

–????? Middleware Vendor Independent API

–????? REST oriented

–????? Asynchronous (Does not mean message passing)

–????? Stateless (Session and Request level)

–????? Idempotent (Process once, Send many)

–????? Everything is uniquely identifiable (URL/URI)

–????? Write Once – Read Many, Don’t Move

–????? Uniform Contract/Interface (event/messages, lookup, and bulk consumers)

–????? Protocol Independent Focus

n? Not coupled to protocol - MFT, HTTP, MQ, FTP, SOAP, JMS

–????? Design for the complexity of the non-happy path - re-run, historical, parallel, errors, etc

–????? Services Expose the how, rather than do.

n? Increase re-use and combination.

n? Declarative vs Imperative

?

Service Oriented Architecture to Microservices

?

n? Service-oriented architecture?(SOA) is a style of?Software design?where services are provided to the other components by?app component, through a?Communication Protocol?over a network. A SOA service is a discrete unit of functionality that can be accessed remotely and acted upon and updated independently, such as retrieving a credit card statement online. SOA is also intended to be independent of vendors, products and technologies.

n? It is self-contained.

n? It is a?black box?for its consumers, meaning the consumer does not have to be aware of the service's inner workings.

n? It may consist of other underlying services.

n? This can be done using anything, but the world has centered on the web.? This means HTTP protocol, Hypermedia (links), and Entities.

n? Further the term Microservices is nothing more than a selection of SOA principles that have emerged and are now commonplace in the industry.? Peter Rodgers introduced the term "Micro-Web-Services “:? --

n? "Micro-Services are composed using?Unix-like pipelines?(the?Web?meets Unix = true?loose-coupling).

n? Services can call services, multiple language run-times.

n? Complex service-assemblies are abstracted behind simple?URI?interfaces.

n? Any service, at any granularity, can be exposed."

n? A well-designed microservices platform "applies the underlying architectural principles of the?Web?and REST services together with Unix-like scheduling and pipelines to provide radical flexibility and improved simplicity in service-oriented architectures.

?

Interfaces

Representational State Transfer (REST) is a style of software architecture for distributed systems such as the World Wide Web.? The REST style is a distillation of key WWW standards and research by Roy Fielding an Innovator and co-founder of the Apache HTTP Server project.

Key goals of REST include: (Align well with ODS objectives)

??????? Scalability of component interactions

??????? Generality of interfaces

??????? Independent deployment of components

??????? Intermediary components to reduce latency, enforce security and encapsulate legacy systems

Principals/Constraints

??????? Client–server – Designed for networked interactions, must be ubiquitous, clients worries about UI.

??????? Stateless – No client context is stored between requests, servers don’t care.

??????? Cacheable – Achieved by Separation of concerns, Stateless, Code on Demand, Uniform Interface

??????? Layered system – Separation of Concerns

??????? Code on demand – Provide code bundles rather than generating presentation server-side

??????? Uniform interface – All requests access the same way, one: protocol, reference, format, etc.

> REST is an approach for Service Orientation - REST is an architectural style applied to implement the service outcomes desired by the service orientation way of thinking.

> REST is about resources/things – In OO objects had behavior and information together.? A resource can be the information of an object, behavior, or anything.? Because the interface only exposes resources object data, behavior, state are all available and consistently referable.

?Please feel free to like , Subscribe and Share my article and provide your feedback. #microservices #dataapi #dataecosystem

?