How These 2 Data Scientists Made It Big in Time of Crisis

I share content about engineering, technology, and leadership for a community of smart, curious people ????. Join my email newsletter and subscribe to my YouTube channel for more updates and tech insights.

“Don’t be afraid to fail. Experience can’t be bought,” Oleg said to the journalist who asked for a piece of advice for entrepreneurs who are starting their journey.

It was late 2020. The Covid-19 crisis brought economies to a standstill. Lots of companies around the world went to the wall or were operating with a skeleton crew. The beginning of the year was also far away from being easy for the young man.

He was working as a software engineer, but was getting tired of programming and started a drop shipping business, which has been a challenge under the pandemic’s circumstances.

But giving up was not an option for him.

In his small town in Ukraine, Konotop, he learned how to become a businessman from an early age. His parents have left their jobs to be personal entrepreneurs and have more control over their lives. Following their steps, he used his savings?—?when he was 12-year-old?—?to buy a lot of school stuff in bulk and stationery like pencils and notebooks, then re-sold them in his mother's shop before the study year.

When he grow up, he graduated from a physics and mathematics school, although he had other interests. Chemistry, electronics, and of course doing business all delighted the boy. He even had his own small laboratory.

Years passed, after trying a bunch of side hustle ventures like indie Android development, cryptocurrency collective mining, and hardware development, which were not always as successful as he wanted them to be, he needed to figure out how to rescue his drop shipping project.

Oleg was worried. He sank in his bed, deep in thought. Then he remembered his friend from the university who spent a lot of time with him and was out of work now. A grin spread across his face.

In that moment, he made his decision and found his?mission.

“Every cloud has a silver lining. This crisis could be an opportunity,” Oleg mumbled to himself.

Without hesitation, he picked up his phone and called his friend.

“Hey Andrew, I think you don’t need to keep looking for a job. I have an idea for a new project. It’s quite a niche but a quintessence of our working experience,” he said passionately.

Andrew was skeptical. After a discussion for about an hour in a Zoom room, he realized that the idea was about a very complex business. Society was moving towards digitization more than ever before. AI and machine learning were on the rise. But there was one thing that would help accelerate things. It was accessing and gathering big data automatically.

“You are a data scientist with a massive experience, and we’re living in the epoch of information. When a lot of people around the globe create billions of terabytes of information every day, we need to analyze it with the proper tools,” Oleg said, trying to convince Andrew.

“Well, you need just to focus on the technical side. I’ll cover all non-technical duties. You can forget the paperwork and business routines,” he added.

The next days, the two young men were sitting at their computers at home and tap-dancing their fingers across their keyboards. They typed a weird pseudo-language full of brackets and dots and semi-colons to build a software that crawls the web?—?yes, the whole web?—?then loads its content, and scrapes data from it.

They hung the browser and cut off its head to empower their product by the headless Chrome browser cluster under the hood.

They implemented a bot that rotates proxies and bypasses CAPTCHA to allow extracting data with ease.

Within a month, they’ve made a fully working Minimum Viable Product (MVP) and immediately released it under the name ScrapingAnt .

ScrapingAnt is a scraping API that offers a large proxy pool with proxies around the world and allows users to scrape all websites without being blocked. It can scrape thousands of pages in parallel and extract data much faster than with the use of single computer power.

After this first release, Oleh Kulyk (Oleg) and Andrew Kovalenko continued to work in an iterative way, which was the best decision they’ve made. It allowed them to be flexible and have better management for their expectations.

Taking a break during tough times to evaluate or re-think their state has helped them as well to follow their goal and stay on their path that was not always as smooth as they wanted it to be. They have struggled with paperwork and business routines before realizing that routines help to control and keep the business organized.

Today, the two young men are 28-year-old and their “Ant” is one of the competitive web scraping tools in the market. They are continuing to improve it and have lately added the residential proxies to it.

The API has helped marketers enhance their strategies and decisions.

But marketers weren’t the only people interested in their product… Business owners, data scientists, machine learning enthusiasts, and many others have also benefited from it.

The Takeaway

At some point in our lives, we may find our decision-making plagued by the ambiguity of the environment. The key here is to acknowledge that volatility, uncertainty, and complexity have a big influence, but we need to persist and try again. With a bit of courage and creativity, we can transform tough conditions into an unexpected gift or a fortunate coincidence.

The two ScrapingAnt founders have noticed the booming of the data industry, especially during the pandemic, and how information became a key for both survival and growth for businesses. They have used their previous working experiences with data mining to build a tool that gives both individuals and companies the possibility to get the right data at the right time efficiently.

In today’s world of cutting-edge technologies, you need to be constantly vigilant and resilient. Remember the skills you already have or you can learn and think about how to leverage them with the current market state. By doing so, you may find a way to implement your potential breakthroughs, even in times of crisis.

You don’t need an endgame in mind to start your big project. Just begin. And if the journey becomes difficult, don’t despair. It means you’re on the right path. The hard one.

Yes, you may fail. But failure is part of the game, and it’s a dress rehearsal for the big day when you finally achieve success. Oleg has recognized that “all the times he failed when he was a child and as an adult taught him more than one success with blind luck would have.” He also considers all his previous trials as important steps that led to the web scraping business.

“Don’t worry about failures, worry about the chances you miss when you don’t even try.”?—?Jack?Canfield

Want more?

???? I share content about engineering, technology, and leadership for a community of smart, curious people. Join my email newsletter and subscribe to my YouTube channel ?? for more insights.

Join 35K+ students and enroll in my Udemy video courses:

• Software Architecture: From Foundations to Cutting-Edge
• Mastering Web Performance: From Novice to Expert