Why Every Engineer and Data Scientist Should Make Every Day a Hack Day

According to Wikipedia, a Hackathon, also known as a Hack Day, is a “design sprint-like event in which computer programmers [and] subject-matter-experts collaborate intensively on software projects”. These Hackathons typically last between a day and a week, and while some are intended simply for educational or social purposes, in many cases the goal is to create actual usable software. 

Hack Days have been a mainstream way for large software companies to achieve multiple purposes. They bring people from different teams and departments together in an attempt to both increase cohesion and inform people about the role of groups one isn’t typically familiar with. In short, it is a great way to connect with others. They are also an attempt for management to advertise their culture and their technology, and create excitement in the workplace, both internally and externally. Organizations known for hosting Hack Days tend to look more appealing to young graduates and professionals, and the argument has been repeatedly made by recruiters to convince potential candidates of the ‘coolness’ and the open-mindedness of their company.

But what does it really mean from an employee standpoint? I can’t speak for everyone, of course, but it seems to me that the general consensus comes to the following: participants like Hack Days because they believe that they offer the possibility to learn new techniques and technologies that they wouldn’t get to touch otherwise. They also get to work on data and problems that are usually beyond their scope. Simply put, Hack Days are great because they force you to “think outside the box”.

As far as I am concerned, I think that Hack Days are exceptional opportunities to grow as a person and as a tech professional. Unfortunately they are hard for companies to organize, and don’t always align with business timelines. There is no reason, however, why you shouldn’t benefit from the “Hackathon mindset” on a daily basis, and I will explain below how you can implement that within your daily responsibilities:

1.   Be open-minded regarding every new project

Your manager comes to you and asks you to work on project B. You are disappointed because you had hoped that you would be assigned to project A, which you were excited about because it always seemed to be the core of the business. If that sounds familiar then keep on reading…

The “Hackathon mindset” says this: be open-minded. Yes, you might not find this new project exciting today. But what if this is because no one else has made it exciting yet? Then it would mean that it is up to you to make it exciting. Don’t think of such projects as dead-ends but as a chance to make a real difference and to pioneer a new area. Keep in mind that most mature projects were often considered the underdog before becoming the “hot” topic. They wouldn't have become hot if everybody had refused to give them a shot.

2.   Don’t be afraid to try something new

Get out of your comfort zone and learn new techniques. It is a common mistake for younger scientists to always try and recycle their knowledge. To some, it might sound reassuring to use a tried-and-true approach that they are familiar with, and they might want to use it again – even unconsciously. This set of mind can be plain dangerous for a researcher, because we are all required to constantly stay up-to-date regarding new technologies, and a new project is the best way to do just that. Technology evolves at a fast pace, new ones are created and others become obsolete regularly.

We also tend to assume that what we have worked on already is our niche, our forte, our “specialty”. However I have personally found out on different occasions, and sometimes with surprise, that a skill that I thought was common among my colleagues was actually not that mainstream, and learning it allowed me not only to take my career to the next level, but also helped my company get ahead in the market. Besides, you might realize that you excel at something you had barely heard of in the first place.

3.   Be ready to work with people you don’t usually work with

I think that virtually everybody nowadays agrees with the fact that people skills are essential to success. Working in solo is overrated, and plain impossible in large companies where many components are necessary to the making of a new product, and many teams are involved. However it is very easy to always default to the same workmates when asking for help, especially if you are a newbie at your company, or if you have a tendency to be shy. The problem with that attitude is that no new collaboration is ever created. I don’t believe that new technologies come from new ideas, but from the combination of multiple visions. Brainstorming with a new group of people instead of conversing with the same handful of colleagues over and over again is key to channeling thoughts into actionable items and hence new opportunities. So just like during Hack Days, go out of your way to hear what other people, especially those you don’t know too well, have to say.

4.   Always assume what you are trying to achieve is feasible

During Hackathons, we are put under pressure to deliver, and deliver fast. Silicon Valley has for a while adhered to the “Fail Fast” mantra, and I only concur with it to some extent. This is because the motto also seems to convey the idea that if something doesn’t work out straightaway, you should just give up on it, and throw the whole concept away altogether. To me, this is a fundamentally flawed thought: you should stand up for your ideas and not fall into what I call the “Wile E. Coyote fallacy”. In the famous cartoon, the Coyote tries to catch the Road Runner with all kinds of sophisticated schemes, systematically fails and moves on to another plan instead of perfecting it. I guess my advice here is to keep things simple, and to be prepared to revise your strategy. It sure is important to edit one’s plans, but equally important to trust one’s original instincts, in order to deliver. You might not achieve the same level of perfection as you originally intended, but your solution can still solve a important business problem and eventually help people.

The short deadline and the sense of urgency experienced on a Hack Day are outstanding ways to get ourselves to accomplish a lot within a short amount of time. Beware of not compromising too much though, as your original ideas are likely to get lost in the process.

5.   Work like there is no tomorrow

I am not suggesting here that you should skip lunch or deny yourself a break (even if I have been guilty of that many times). I am simply saying that the capacity to fight the urge of leaving things to tomorrow with no valid excuse is an invaluable asset in somebody’s career. Procrastinating is the best way to ensure things will never get done. So go ahead and hack a quick solution even if you feel that it’s not the most efficient, elegant and robust one. In the worse case scenario it will give you a sense of how complex your idea/project really is, and what resources will be necessary in the future. Don’t get obsessive about how your code looks and how to optimize it.

Also, do not wait too long for people to get back to you. You should definitely ask for help, especially if you know of someone who can point you to the right data, or to some pre-existing code or work that you could use as a starting point. However, if you emailed someone to request some guidance, and that person didn’t get back to you even after you have seen him/her at the coffee machine, it might be your cue to look for another way to access the information you need.

6.   Don’t assume that someone has done it before

It is frequent to assume that simple ideas have no merit, because “someone must have thought of that before”. Because a lot of people think that way, some ideas are just being dismissed for appearing too obvious or too straightforward at first. This is especially true in the context of very large companies. I have personally encountered a situation where I thought of using an algorithm that was mainstream in Particle Physics (my original field of expertise), but then assumed that someone else must have tried that already since there are many other particle physicists in Data Science. So I didn’t look into it any further. Yet I eventually found out months later that no one had actually attempted to use that technique outside of Physics...

Experience has also taught me that even old ideas can lead to a wildly different execution. I am yet to find a project where a fresh take on an existing product didn’t at least add a new dimension to it. Your work will never be as useless and pointless as it might first seem to you, as long as you do it with conviction.

7.   Research your problem before you start

The first step during a Hack Day should be to research possible publications regarding your project. You should do the same whenever you start working on something new. What else was attempted? What is the common way to solve similar problems? That means that Google is your best friend when you try to get a sense of what both the industry and academia have to say about a specific subject. Even blog entries can contain interesting inputs as long as you stay critical of what you read. It is that very process that keeps R&D alive and allows theoretical work from university labs to be applied in industrial settings. Staying informed is key to the ability to quickly assess the situation and avoid false starts.

8.   The end justifies the means…

… even if the means are not the ones you would like them to be.

This is an issue I am particularly sensitive to. Over the past few years, I have heard too many people say they are interested in working with a specific technique. I will give an example that I think speaks volumes about one of the biggest misconceptions regarding how Data Science should work. One of the trends in Machine Learning nowadays is without a doubt Deep Learning. The “Deep Learning movement” is getting a lot of momentum among both techies and business people. Yet it is crucial that Machine Learning professionals should constantly keep in mind that the output matters more than the technology used.

I am naturally all in favor of spreading new technologies to the largest crowd possible, so that people know what tools exist on the market, know how they work, and more specifically when to use them. But while Deep Learning is a great way to solve some problems, starting a project with the ultimate goal of using this one specific technique at all costs is limiting and narrow-minded. Sometimes a simple regression goes a very long way and picking whatever algorithm is the most appropriate is the way to go, period.

9.   Don’t assume people understand everything you say

When you hack something new, be ready to share it with both tech and non-tech people. Your work will be worthless if people outside your inner circle of colleagues cannot understand a word you say. Prepare a demo of your product as well as documentation that goes along with it, so that everyone, no matter their background, understands what your product is, what is at stake, and what your idea means for the business. Getting things lost in translation is one of the biggest enemies of data scientists, because for many people outside the Tech community, Data Science has more to do with black magic than it does with science. Therefore people tend to have unrealistic expectations, or not see the real value that lies behind data, code and technology. Time has come for us to break the tech language barrier, and like we would on a Hack Day, share our work with other professionals from other fields.

10.  Always assume people will reuse your work

Finally, if you have come this far in this article, you are likely an optimistic professional who believes that my point of view has merit. Therefore you must also believe that there is no reason why, should you follow the previous steps, your work would not lead to new products and business opportunities. At this point it probably comes to you as self-evident that you should make your code readable and understandable by fellow engineers and scientists, reusable as much as possible, and well-documented. Even if your code is not production-ready, it will make it easier for team members to take it to the next level, provided you kept that in mind while working on your new project.

                                                                                         -- Jennifer Prendki