Lessons Learned From My First Kaggle Competition

As a data scientist, you're probably heard an endless amount of talks from other YouTube regarding the data science journey about doing the Titanic Kaggle challenge. I've personally avoided Kaggle for a long time, but recently I had the chance to help coordinate a Kaggle competition for Brambles Data Scientists. I wanted to share a little bit about what I learned from it and some lessons that I think are relevant for others who want to implement something like this in their own company.

What is Kaggle?

Kaggle is a website that offers a multitude of things, but the most famous thing that they are known for is hosted competitions that have monetary prizes. Kaggle competitions are varied in size and scope, but a lot of companies will host competitions, with some names like Google, AMEX, H&M have hosted competitions. Companies get great creative solution while a specific team or individual will be able to get cash for their winning contribution.

While Kaggle is most known for their competitions, they actually have a few other valuable things that they provide: from education (most famous is the Titanic tutorial) to a rather robust integrated Jupyter notebook that you can do initial experiments on with the Kaggle datasets that are provided. All in all, Kaggle is an excellent way to be able to start your data science journey with dataset that have been commonly explored.

Journeying into the Kaggle Forest with the Team

A few months ago, I worked with my manager to host our first team Kaggle competition. We picked the H&M Personalized Fashion Recommendations competition, broke off into teams and then begun journeying to make predictions against the dataset. For teams, we broke off into three teams based upon the competitions rule limits, then timezones, and varying experiences.

From there each team would discuss their preferred method of machine learning model and then begun working on the algorithm. For the first competition, I was assigned as a project manager/product owner/scrum master - someone who was overseeing the other teams and ensuring that we were moving along with the competition and making sure that we were submitting entries into the team.

Every week we would meet together and run through our status updates, run through some demo work, and even do some training from academic system. In some ways we treated the meeting like a standup meeting for the competition - identifying any needs or roadblocks that we needed to get removed before moving on to the next milestones.

We jumped into this competition with about a month left to go and were able to submit at least three submissions into Kaggle. Spoiler alert, we didn't win; not even close! But we came out of it with some interesting discoveries.

Some Lessons After Completing One Kaggle Competition

1. Collaboration is important outside of your own product squad

In agile (and/or the Spotify model), something that I feel like I miss from being in the business world is working with other colleagues within a specific department. Being product centric, I interface pretty regularly with other data scientists who are working on the same product, but outside of that, it was rare to work with my other data science colleagues.

By getting all the scientists together and making teams, we were able to cross-pollinate ideas and be able to share some of the knowledge of what we were working on and even get a fresh perspective on how to address/tackle a data science problem. In one of the weekly standup calls that we had, we were looking at the data from the H&M competition and we began debating whether or not it was worth it to use a certain feature or not. My team had been thinking about using it while another was debating against it. Being able to challenge each other's perspectives was key to

2. Expanding the knowledge of Data Science, especially outside of your expertise

One of the reasons why I picked the H&M Kaggle competition as our first one was because of the algorithm that was being suggested. Our line of work has not found any use cases for recommender systems and thus, I had little to no experience in using it. I thought it would be interesting to get a bunch of people who have a varying range of experiences in recommenders to see if we could find a different way to tackle the problem.

Ironically, the recommender system has come up in some of the discussions with our product stakeholders. Being put outside of your wheelhouse allows you to work on a new problem, but more importantly, ask how can this algorithm be transferred back into our business. Even more importantly, it's how it can assist and make business decisions. Being familiar with the algorithm through the Kaggle competition helped me be able to articulate the method and then put ideas/business actions behind the algorithm to be able to sell it to key stakeholders.

3. Culture is Key

Just like any kind of professional development, a Kaggle competition requires a lot of time and effort. For us, my company instilled a department/company-wide initiative called "Focus Fridays". It's where for us we are allowed to focus on personal/professional development. It also is a time and place where meetings are kept to the bare minimum.

This was largely helpful for us because this guaranteed us one day a week where we were able to meet up, work on the Kaggle competition and be able to make incremental progress on our work. The fact that this wasn't extracurricular that was required outside of normal working hours is so refreshing, especially in a tech culture where many people have to work on personal/professional projects or development outside of the working hours. While working on this project once a week won't win you any awards, it certainly helps you with getting to know the problem, working on it to be able to create a solution, and possibly do a bit of tuning on the algorithm.

In conclusion, I think there's plenty of reasons why you should get people to work on a Kaggle competition from work.

If you like what you're seeing from this article and think, I want to work for a company like that... Brambles is actually hiring! DM me if you want to chat more or visit the CHEP website for openings.