Case study : Predictive Maintenance in Manufacturing by leveraging Databricks

Birlasoft | Databricks

Background:

A manufacturing company is grappling with significant business challenges, including frequent production delays and escalating maintenance costs. Machine breakdowns are becoming persistent issue, severely impacting operational efficiency. The company is struggling to identify early signs of equipment failure, leading to unexpected downtime and costly repairs. Despite efforts to improve, they lack the predictive insights needed to mitigate risks and maintain smooth production flow. The inability to address these problems is hindering ability to meet production targets, affecting overall profitability. Despite need of a solution to enhance machine reliability and reduce unplanned maintenance expenses.

What are the business problems?

1. Production delays and missed deadlines
2. High maintenance costs due to reactive repair work
3. Poor asset utilization, as equipment is under-maintained or over-maintained
4. Reduced product quality because of equipment inefficacies
5. Safety risks due to unexpected breakdown

Solution Scope:

The need is for a solution that can predict potential equipment failures in advance reducing downtime, optimizing maintenance schedules, and improving overall operational efficiency

Implementation coverage:

1. Data collection from various sensors, machine logs, and historical maintenance data

2. Data processing to clean and transform the raw data for predictive model building

3. Development of machine learning models to predict equipment failure and estimate remaining useful life (RUL) for critical machinery

4. Real-time monitoring of equipment health

5. Proactive maintenance scheduling based on failure predictions

How Databricks tools addresses these business problems?

Data integration:

Databricks simplifies the ingestion of real-time sensor data and large volumes of historical data

Using Delta Lake, all types of data (structured, semi-structured, unstructured) are seamlessly stored and managed

With Databricks Auto loader, real-time streaming data from equipment sensors is processed as soon as it arrives

Data Preprocessing and Feature Engineering:

Databricks notebooks allow engineers to quickly build and test data pipelines for cleaning and preparing data from multiple sources.

Used Advanced feature to identify key patterns in sensor data (vibration, temperature, pressure) that indicate early signs of equipment failure.

Predictive Maintenance Models:

Machine Learning (ML) in Databricks is used to build and train models for predictive maintenance. Databricks MLflow helped to provide seamless integration for experiment tracking, model versioning, and model deployment.

Databricks allowed to use deep learning and time series forecasting to predict failures by analyzing historical breakdown patterns and sensor data trends.

Real-Time Monitoring & Alerts:

Databricks Structured Streaming enabled to have a continuous monitoring of machine performance in real-time, detecting anomalies and triggering alerts for maintenance teams.

Predictive models were deployed on streaming data to forecast the likelihood of a breakdown and send automatic notifications to the relevant personnel.

Scalability and Collaboration:

Databricks efficiently scaled with the growing amount of sensor data as more machines are added, while providing a collaborative platform for data scientists, engineers, and business analysts to work together.

Using Delta Sharing, able to share insights shared securely to stakeholders, maintenance teams, operations, and management.

Business Benefits:

Reduced Downtime: Proactively scheduling maintenance based on predicted failures ensures that machinery is serviced before any disruption, leading to a significant reduction in unplanned downtime (target KPI: > 90% reduction).

Lower Maintenance Costs: Maintenance becomes more efficient, shifting from reactive repairs to condition-based maintenance, reducing labor and parts costs by 30-40%.

Increased Equipment Lifespan: Predictive maintenance extends the lifespan of machinery by preventing wear and tear caused by neglected repairs, boosting overall asset utilization.

Enhanced Productivity: Improved uptime increases production efficiency by 10-20%, ensuring that production deadlines are met, and throughput is maximized.

Improved Safety: Early detection of equipment issues reduces the likelihood of catastrophic failures, thus improving workplace safety and reducing the number of incidents (target KPI: 50% reduction in incidents).

Better Decision-making: Real-time analytics provide operational insights, helping teams optimize production schedules and further reduce costs.

Databricks features leveraged:

Delta Lake, Databricks Auto Loader, Structured Streaming, Apache Spark, Databricks Notebooks, MLflow, Databricks Runtime for ML, Delta Sharing, Databricks SQL, Databricks Repos Version Control, Azure integrations, Job scheduling with Databricks Jobs, and Power BI visualizations

Deployed Architecture:

1. Data Ingestion Layer (Bronze)

Source Systems: Upstream source systems generates and places Sensor, Machine log data Azure ADLS.

Mount Points: Used Azure Data Lake Storage (ADLS) mount points in Databricks to access raw data.

Storage Credentials: Established secure access using Azure Key Vault for storing secrets and credentials.

2. Data Transformation Layer (Silver)

Data Cleaning: Performed data cleaning operations such as removing duplicates, handling missing values, and standardizing formats.

Transformation: Created new columns, aggregate data, and apply business logic.

Intermediate Storage: Store the cleaned and transformed data in the silver layer of the medallion architecture.

3. Data Enrichment Layer (Gold)

Business Logic: Applied filters, transformations for further enrich the data for specific business use cases.

Aggregation: Aggregated data to create summary tables and metrics.

Storage: Saved the enriched data in the gold layer, ready for reporting and analysis.

4. Data Consumption Layer

Power BI Integration: Loaded the Gold layer data into Power BI for creating business decision-making reports.

Dashboards and Reports: Developed interactive dashboards and reports in Power BI to visualize the data and derive insights.

Detailed Components

1. Azure Data Lake Storage (ADLS)

Mount Points: Configured mount points in Databricks to access ADLS.

Security: Used Azure Key Vault to manage and access storage credentials securely.

2. Databricks

Data Cleaning: Used Databricks notebooks to clean and preprocess data.

Transformations: Implemented transformations using PySpark or SQL in Databricks.

Medallion Architecture: Organized data into bronze, silver, and gold layers.

3. Azure Key Vault

Secrets Management: Store and manage secrets, such as client secrets and storage account keys, securely.

4. Power BI

Data Loading: Connected Power BI to the Gold layer in ADLS for data refreshing

Report Creation: Created reports and dashboards to visualize the data.