Temporal Fashion Recommender

Building a Recommender That Evolves with Seasons

One of the biggest challenges in the design of any recommender system (RecSys) is in handling temporal shifts. Since the world of fashion evolves with time, recommenders we design must adjust to the changing tides as well.

In this article, we will consider how we can factor in the seasonal outfit subtleties into our Convolutional Neural Network (CNN) model using temporal weights.

The Business Problem

Our imaginary eCommerce company, HappyPanda Co., is expanding globally and will like to launch Seasonal Collections. The product requirements are:

• Given collections of outfits, how can we adjust them to suit the various seasons across different countries? Note that each country will have different fashion sense and weather conditions for different seasons.

The Data

• We will be using a subset of DeepFashion data open-sourced by Liu Z. et al., The Chinese University of Hong Kong. Our data consists of 280K fashion images across 46 categories. You can download the data from their website. Furthermore, the team has released an updated version with additional data. You will need to fill up a Google form to gain access to the data.
• The temperature data is from Kaggle — Climate Change: Earth Surface Temperature open-sourced by Berkeley Earth. The data consists of global temperatures across 3,400 cities from 1750 to 2013. We will only be using a subset of the data, the average temperature of Madrid and Melbourne across 10 years (2004–2013).

From the deep fashion data, I select outfits suitable for each of the four seasons. For example, in winter, most people will don on a heavier jacket for warmth. On the other hand, spring and summer is the time to enjoy some time under the sun with lighter clothes. Temperature thus plays a significant role in influencing what people wear. For instance, if we consider the average temperature across the year in a particular city, Madrid (Spain), we can place our selected outfits across time to observe the relationship between outfits and temperature.

More importantly, the transition between the seasons takes place over weeks, and our model should be able to adjust the recommendations over time, helping our users transit into the next season gradually.

Of course, in the real world, we will have access to a richer set of time series data. For example, customers’ view/purchase trends over time, hot sellers, or fads like Korean Kpop Fashion. Weather data here is an illustration of the potential of incorporating time series data and not the definitive nor sole data solution for the problem.

Reviewing Results from Seasonal Collection

Before we dive into the code, let us take a look at the input/output of the model to gain intuition on the problem and proposed solution.

Input:

• I select 24 outfits for each season. They will act as the seed images for Seasonal Collections.
• I will only pass images into the model; the model does not ingest additional attributes/descriptions/product details; this is because we want the model to automatically learn and detect the style of fashion images passed into it without further human/machine labeling.

Output:

• The goal is for Seasonal Collections to blend in the influence of each season, weighted by how deep into the season we are.

Madrid’s weather transits in a gradual manner from Winter to Spring from January to May. Seasonal Collections can generate a new collection for the transition period by blending the features of winter (coat, long sleeves, full-length wear) with features from Spring (lighter materials, colorful wear, less layering) into a new “Spring is coming collection”. The ratio we use here is 50–50, so both Winter and Spring have equal weight in the final blend (we will explore this concept in the code section).

In contrast to Madrid, Melbourne’s Summer transits to Autumn rapidly, with temperatures falling several °C every week. Thus, we need to aggressively introduce elements from the Autumn collection to match our customers’ requirements for warmth. Using a blending ratio of 25–75, we assign a higher weight to the Autumn collection, allowing our model to pull out essential attributes such as layering, long sleeves, and thicker materials to keep our customers warm.

We can see the importance of localization; to tune the model for different cities across different periods. There is no one size fits all solution for RecSys, especially when we want to recommend across temporal data.

The Code

Link to code on Colab (you just need a free Google Account to run the code on GPU in the cloud).

Seasonal Collections is built on Convolutional Neural Networks, with transfer learning from ResNet and approximate nearest neighbors. I will not dive into the individual theories as they have been well covered in many articles:

Explanation of Overall Setup:

• Building a Personalized Real-Time Fashion Collection Recommender
• Transfer Learning with Convolutional Neural Networks in PyTorch

Convolutional Neural Networks:

• Convolutional Neural Networks (CNNs) explained by deeplizard (Video)
• Convolutional Neural Networks (CNN) Introduction
• A Beginner’s Guide To Understanding Convolutional Neural Networks

ResNet and Transfer Learning:

• Understanding Deep Residual Networks
• An Overview of ResNet and its Variants
• A Comprehensive Hands-on Guide to Transfer Learning with Real-World Applications in Deep Learning

Approximate Nearest Neighbors:

• Erik: Annoy: Nearest neighbors and vector models — part 2 — algorithms and data structures

Instead, I will focus on the core logic of Seasonal Collections:

The command to get the list of outfits for the Melbourne example above is this line of code:

get_similar_image(generate_image_mix(summer_ids, autumn_ids, number_of_items=24, ratio=0.25))

• In essence, we are taking two collections (summer_ids and autumn_ids) and mixing them up based on the ratio (0.25) and selecting 24 items from the mix.
• This means that 6 outfits from summer and 18 outfits from autumn will be used as the seed to generate the new collection.
• The ratio is thus the critical hyperparameter that can be tuned based on the time-series pattern. If we want to increase the influence of a collection, we tune the ratio in order to include more outfits of that collection in the mix.
• Centroid embeddings will then merge all the seed images into one representation by averaging the values across all dimensions. Finally, approximate nearest neighbors is applied to return outfits closest to the representation.
• We can place the get_similar_image() code fragment above in a loop and generate a mix of collections based on different ratios:

Ratio 1.00 Winter vs. Spring. Outfits from DeepFashion, open-source by Liu Z. et al.

Ratio 0.75 Winter vs. Spring. Outfits from DeepFashion, open-source by Liu Z. et al.

Ratio 0.50 Winter vs. Spring. Outfits from DeepFashion, open-source by Liu Z. et al.

Ratio 0.25 Winter vs. Spring. Outfits from DeepFashion, open-source by Liu Z. et al.

Ratio 0.00 Winter vs. Spring. Outfits from DeepFashion, open-source by Liu Z. et al.

What have we learned

We can intuitively incorporate seasonality and time-series data into RecSys by tweaking the ratio to blend the different collections. However, it is not easy to determine this ratio in the real world. We will need to conduct a lot of AB testing, and gather feedback from users, domain experts, product teams. Furthermore, we will have to localize the solution across different geographies and markets to cater for the difference in fashion trends and even the weather.

Explore the rest of Modern Visual RecSys Series

1. How does a Recommender Work? [Foundational]
2. How to Design a Recommender? [Foundational]
3. Intro to Visual RecSys [Core]
4. Convolutional Neural Networks Recommender [Pro]
5. COVID-19 Case Study with CNN [Pro]
6. Building a Personalized Real-Time Fashion Collection Recommender [Pro]
7. Advanced topics — Temporal Modeling [Pro][we are here]
8. The Future of Visual Recommender Systems: Four Practical State-Of-The-Art Techniques [Foundational]