Assortment Planning and Optimization with AI

Executive Summary

?

In the rapidly evolving retail landscape, effective assortment planning and optimization are crucial for retailers to meet customer demands, maximize profitability, and maintain a competitive edge. This article presents a comprehensive exploration of an advanced assortment allocation system designed to address the complexities of modern retail. By integrating sophisticated machine learning techniques, modeling cannibalization effects, handling new products without historical sales data, and implementing an iterative optimization process, the proposed solution offers a robust framework for optimizing product assortments.

?

Key Highlights:

• Advanced Demand Forecasting: Utilizing machine learning models with advanced feature engineering to capture seasonality, trends, cannibalization effects, and estimate demand for new products.
• Modeling Cannibalization Effects: Incorporating cross-elasticity coefficients and constraints into both demand forecasting and optimization models.
• Handling New Products: Leveraging similarity models, such as CLIP, to estimate demand for new products without historical sales data.
• Iterative Optimization Process: Implementing a feedback loop where demand forecasts and assortment decisions inform each other until convergence.
• Enhanced Optimization Model: Building a robust model using Pyomo, considering inventory, capacity, cannibalization constraints, and the inclusion of new products.
• Scalability and Performance: Employing high-performance libraries and parallel computing to handle large datasets efficiently.
• Validation and Evaluation: Implementing methods for model validation, performance measurement, and simulation to assess effectiveness.
• Ethical Considerations: Addressing data privacy, algorithmic bias, and responsible AI practices.
• Integration into Retail Operations: Providing best practices for integrating the system into existing workflows.
• Business Benefits and ROI: Highlighting the expected return on investment and impact on gross margin.

Code example at the end of article

Introduction

In today's fast-paced retail environment, effective assortment planning and optimization are more critical than ever. Retailers face the challenge of offering the right mix of products to meet customer demands while maximizing profitability. This task becomes even more complex with the introduction of new products lacking historical sales data and the need to account for cannibalization effects among similar products.

?

This article presents a comprehensive exploration of building an advanced and sophisticated assortment allocation system. We dive into the challenges of demand forecasting at a granular level, modeling cannibalization effects, handling new products without prior sales data, optimizing product allocation, and ensuring scalability and performance. The solution integrates cutting-edge machine learning techniques, similarity models, iterative optimization processes, and robust data processing methodologies.

Problem Context

Challenges in Assortment Planning

1. Demand Forecasting: Predicting product demand at the store level is essential for effective assortment planning. Forecasts must account for seasonality, trends, promotions, cannibalization effects, and the uncertainty associated with new products lacking historical data.
2. Cannibalization Effects: Introducing similar products can lead to cannibalization, where one product's sales reduce the sales of another. Properly modeling these effects is crucial to avoid overstocking and lost sales opportunities.
3. New Product Introductions: Forecasting demand for new products without historical sales data poses a significant challenge. Traditional forecasting methods may not suffice, requiring innovative approaches like similarity modeling and leveraging product attributes.
4. Inventory and Capacity Constraints: Stores have limited capacity, and products have finite inventory levels. Allocations must respect these constraints to prevent stockouts and overstocking.
5. Complex Interdependencies: The assortment offered influences demand, and demand influences assortment decisions. Capturing this cyclical relationship adds complexity to the modeling process.
6. Scalability and Performance: Handling large datasets and complex optimization models requires efficient algorithms and scalable data processing techniques.

Approach Overview

To address these challenges, we adopt a comprehensive approach that includes:

1. Advanced Demand Forecasting: Using machine learning models (e.g., XGBoost) with advanced feature engineering to capture seasonality, trends, cannibalization effects, and estimate demand for new products.
2. Modeling Cannibalization Effects: Incorporating cannibalization effects into both demand forecasting and optimization models, using cross-elasticity coefficients and constraints.
3. Handling New Products: Leveraging similarity models, such as CLIP (Contrastive Language-Image Pre-training), to estimate demand for new products without historical sales data.
4. Iterative Optimization Process: Implementing an iterative process where demand forecasts are adjusted based on the proposed assortment, and the assortment is re-optimized accordingly.
5. Enhanced Optimization Model: Building a robust optimization model using Pyomo, considering inventory, capacity, cannibalization constraints, and the inclusion of new products.
6. Scalability and Performance Enhancements: Leveraging high-performance libraries like Polars for data processing and utilizing parallel computing for efficiency.
7. Validation and Evaluation: Implementing methods for model validation, performance measurement, and simulation to assess effectiveness.
8. Ethical Considerations: Addressing data privacy, algorithmic bias, and responsible AI practices.
9. Integration into Retail Operations: Providing best practices for integrating the system into existing workflows and ensuring user adoption.

Solution Details

1. Data Generation with Enhanced Realism

Synthetic Data Creation

To simulate a realistic retail environment, we generate synthetic data that includes:

• Stores: A set of retail stores with varying capacities.
• Products: A collection of products, including both existing and new products, each assigned to a product category.
• Product Attributes: Features such as images, descriptions, promotions, and trends.
• Sales Data: Historical weekly sales data over two years, incorporating seasonality and trend components for existing products.
• Inventory Levels: Current inventory levels for each product.
• Size Distribution: Sales data at the size level to capture granular demand patterns.

Incorporating Seasonality and Trends

We introduce seasonality and trends using sine functions and linear trends to mimic real-world sales patterns. This enhances the model's ability to capture temporal variations in demand.

2. Advanced Feature Engineering and Demand Forecasting

Feature Engineering

Advanced feature engineering is crucial for improving forecasting accuracy. Key features include:

• Categorical Encoding: Transforming categorical variables (e.g., store, product, category) into numerical codes.
• Date Features: Extracting features like day of the week, day of the year, week of the year, month, and year.
• Seasonality: Creating sine and cosine transformations to model seasonal patterns.
• Lag Features: Including lagged sales values (e.g., sales from previous weeks) to capture autocorrelation.
• Rolling Statistics: Computing rolling means to capture trends over time.
• Assortment Features: Adding features that represent the assortment composition, such as the number of similar products available.

Machine Learning Model: XGBoost

We use XGBoost, a gradient boosting algorithm, for demand forecasting due to its ability to handle nonlinear relationships and interactions. The model is trained on the engineered features to predict future sales.

Time Series Cross-Validation

To validate the model's performance, we employ time series cross-validation, which respects the temporal order of data and provides a more realistic assessment of forecasting accuracy.

Incorporating Cannibalization Effects

Modeling Cannibalization in Demand Forecasting

To capture cannibalization effects in demand forecasting:

• Cross-Elasticity Coefficients: Estimating how the demand for one product is affected by the availability of similar products.
• Assortment Variables: Including features that represent the presence of similar products in the assortment.
• Adjustment of Demand Forecasts: Modifying demand forecasts based on cross-elasticity coefficients to reflect cannibalization.

Incorporating Cannibalization in Optimization Model

In the optimization model, we:

• Define Cannibalization Constraints: Limit the number of similar products allocated to each store to prevent excessive cannibalization.
• Penalize Cannibalization in Objective Function: Optionally, include penalties for assortments that may lead to high cannibalization.

Handling New Products Without Historical Sales Data

Challenges with New Products

New products lack historical sales data, making it difficult to forecast demand using traditional time series methods. Accurately estimating demand is crucial to avoid overstocking or understocking and to make informed assortment decisions.

Leveraging Similarity Models

We use similarity models to estimate demand for new products:

• CLIP Model: Utilize the CLIP model, which creates embeddings for images and text, mapping them into a shared vector space.
• Product Embeddings: Extract embeddings for new and existing products using their images and descriptions.
• Similarity Calculation: Compute cosine similarity between the embeddings of new products and existing products.
• Estimating Demand: Use the historical sales data of similar existing products, weighted by similarity scores, to estimate demand for new products.

Integration into the System

• Feature Engineering Pipeline: Include embeddings and similarity scores in the feature set used for demand forecasting.
• Demand Forecasting Adjustments: Modify the forecasting model to handle new products without historical sales data.
• Optimization Model Updates: Ensure that the optimization model includes new products and adjusts constraints accordingly.

Optimization Model

Optimization Framework

We build the optimization model using Pyomo, an open-source optimization modeling language in Python. The model aims to maximize total profit while considering various constraints, including the inclusion of new products with estimated demand.

Decision Variables

• Allocation Variables: The number of units of each product allocated to each store, including new products.

Objective Function

• Maximize Total Profit: Calculated as the sum of the gross margin per unit multiplied by the allocated units, adjusted for cannibalization effects.

Constraints

1. Demand Constraints: Allocations cannot exceed the adjusted demand forecasts, including estimated demands for new products.
2. Inventory Constraints: Total allocations of a product cannot exceed its inventory level.
3. Capacity Constraints: Total allocations to a store cannot exceed its capacity.
4. Cannibalization Constraints: Limits are set on the number of similar products (from the same category) allocated to a store.
5. Uncertainty Consideration for New Products: Optionally, include conservative allocation limits for new products due to higher demand uncertainty.

4. Iterative Optimization Process

Recognizing the interdependence between demand and assortment, we implement an iterative process:

1. Initial Demand Forecast: Generate initial forecasts, including estimated demands for new products.
2. Optimize Assortment: Use the initial forecasts to optimize the assortment.
3. Adjust Demand Forecasts: Update forecasts based on the proposed assortment, incorporating cannibalization effects.
4. Re-optimize Assortment: Use the adjusted forecasts to re-optimize the assortment.
5. Convergence Check: Repeat steps 3 and 4 until the allocation plan stabilizes.

This iterative approach ensures that both demand forecasts and assortment decisions are aligned and reflect the impact of cannibalization and the uncertainty of new products.

Scalability and Performance Enhancements

Data Processing with Polars

We use Polars, a high-performance DataFrame library, for efficient data processing. Polars leverages Apache Arrow memory formats and is optimized for speed, making it suitable for large datasets.

Parallel Computing

Parallel processing is utilized to speed up computations, especially when adjusting demand forecasts, processing embeddings, and during the iterative optimization process. Libraries like joblib are used for easy parallelization.

Efficient Solvers

For solving the optimization model, we use efficient solvers like GLPK or Gurobi (if available). These solvers can handle large-scale optimization problems effectively.

Performance Optimization for Embeddings

Processing embeddings for a large number of products can be computationally intensive:

• Batch Processing: Process embeddings in batches to utilize hardware acceleration efficiently.
• Caching Embeddings: Cache embeddings of existing products to prevent redundant computations.

Validation and Evaluation

Model Validation

• Cross-Validation: Employed during demand forecasting to assess model performance.
• Backtesting for New Products: Simulate the introduction of past new products to validate the demand estimation method.
• Holdout Testing: Using a separate dataset to test the model's predictive accuracy.

Performance Metrics

• Mean Absolute Error (MAE): Used to measure the accuracy of demand forecasts.
• Total Allocated Units and Expected Profit: Calculated after optimization to evaluate the effectiveness of the allocation plan.

Simulation and What-If Analysis

Simulation is conducted to assess the impact of demand uncertainties on profit:

• Demand Simulation: Adjusting demand forecasts by introducing random variations, especially for new products with higher uncertainty.
• Re-optimization: Running the optimization model with simulated demands.
• Analysis: Calculating the average profit and standard deviation to understand the potential range of outcomes.

Monitoring and Feedback Loop

• Post-Launch Monitoring: After launching new products, compare actual sales with forecasts.
• Model Adjustments: Update models based on actual performance to improve future forecasts.

Ethical Considerations

?

Maintaining ethical standards is crucial for building trust and ensuring the responsible use of AI in assortment planning.

• Data Privacy and Security: Objective: Protect sensitive customer and business data. Approach: Implement data anonymization, encryption, and compliance with regulations like GDPR and CCPA. Benefits: Safeguarded data integrity and compliance with legal standards.
• Algorithmic Fairness: Objective: Prevent biases in assortment decisions. Approach: Employ fairness metrics, diverse training data, and transparency in model operations. Benefits: Equitable assortment decisions and enhanced stakeholder trust.
• Transparency and Explainability: Objective: Make AI-driven decisions understandable to stakeholders. Approach: Utilize interpretable models and provide clear explanations for assortment recommendations. Benefits: Increased confidence in AI systems and better stakeholder alignment.

Integration into Retail Operations

Best Practices for Integration

• Stakeholder Engagement: Involve key users and stakeholders early in the development and implementation process.
• API Development: Create APIs for seamless integration with existing systems (e.g., ERP, POS).
• Modular Deployment: Implement the system in phases to minimize disruption.
• Data Integration: Ensure compatibility with existing data formats and databases.
• Training Programs: Provide comprehensive training and support for users.
• Change Management: Use strategies to facilitate adoption and address resistance to change.

User Experience Considerations

• Intuitive Interfaces: Design user-friendly dashboards and tools for interacting with the system.
• Visualization Tools: Provide visual aids to help users understand data and insights.
• Feedback Mechanisms: Implement features that allow users to provide feedback and report issues.

Business Benefits and ROI

?

Further analysis of the system's impact will demonstrate its value and support strategic investment decisions.

• Comprehensive ROI Metrics: Objective: Quantify the financial benefits of the assortment planning system. Approach: Track metrics such as sales growth, inventory turnover, gross margin return on investment (GMROI), and stockout rates. Benefits: Clear evidence of the system’s effectiveness and justification for continued investment.
• Case Studies and Real-World Applications: Objective: Showcase practical implementations and success stories. Approach: Develop case studies demonstrating the system's application in various retail scenarios. Benefits: Illustrative examples that highlight the system's capabilities and benefits.

?

Expected Benefits for Fashion Retailers

• Increased Sales: Better assortment planning leads to higher customer satisfaction and increased sales.
• Improved Gross Margins: Optimized inventory reduces markdowns and stockouts, improving profit margins.
• Inventory Efficiency: More accurate demand forecasting reduces excess inventory and associated holding costs.
• Competitive Advantage: Advanced analytics provide insights that can differentiate retailers in the market.
• Customer Loyalty: Enhanced product availability and selection improve customer experience and loyalty.

Return on Investment (ROI) and Gross Margin Impact

• ROI Estimates: Industry studies suggest that advanced assortment planning can improve sales by 2-7% and reduce inventory costs by 5-10%.
• Gross Margin Improvement: Margins can improve by up to 5% due to reduced markdowns and optimized pricing strategies.
• Payback Period: With significant cost savings and revenue increases, the investment in such a system can often be recouped within one to two years.

Performance Measurement

• Key Performance Indicators (KPIs): Sales growth Inventory turnover rates Gross margin return on investment (GMROI) Stockout rates Customer satisfaction scores
• Regular Reporting: Generate periodic reports to track KPIs over time and make data-driven decisions.

?

Conclusion

The advanced assortment planning and optimization system presented in this article addresses the complex challenges faced by retailers in today's dynamic environment. By integrating advanced machine learning techniques, modeling cannibalization effects, handling new products without historical sales data, and implementing an iterative optimization process, the solution provides a robust framework for maximizing profitability while respecting operational constraints.

Key Takeaways:

• Holistic Approach: The system offers a comprehensive solution that integrates multiple facets of assortment planning.
• Scalability and Performance: Designed to handle large datasets and complex models efficiently.
• Adaptability: Capable of adjusting to changes in market trends, consumer behavior, and product offerings.
• Ethical and Responsible: Incorporates ethical considerations to ensure responsible AI practices.

By adopting this comprehensive approach, retailers can make informed assortment decisions that align with customer preferences, minimize cannibalization, effectively introduce new products, and drive business growth.

Source Code