The future of Data Science

I started working with Machine Learning and Data Science a few years back. I am not the most experienced in the field, nor the best, but I have spent a lot of time thinking about how the field would evolve.

What is happening!?

I have observed several initiatives attempting to automate the process of creating machine learning frameworks. Take Auto-SkLearn for instance -it is great. It helps you find the right method for constructing your framework. It makes the machine learning world more accessible for everyone. People don’t need to get a Master’s degree or spend hours on Coursera to use artificial intelligence.

I’m not trying to invalidate those methods; by all means, I think education transforms lives. However, I do believe that making artificial intelligence available for a large portion of the population is contributing not only to societal development, but also to the artificial intelligence community as well

In the past, brilliant mathematicians created advanced methods to solve problems; things like Neural Networks, Support Vector Machine, and others have been around for decades. The only problem is that the advanced methods lacked two things that they needed to thrive: large amounts of data and computational power.

Now comes the big data era, which makes data collection, storage, and availability easier than ever before. Sensors, cameras, and other connected devices provide data on a large scale and at great speed. Computational power is cheaper - I have a 32 core, 128GB RAM computer at home, and I paid less than $1k for it. Now it is time for the methods created in the past to thrive, as they never have before. More people are being educated in a job market that 5 years ago no one could imagine, and more initiatives like the autoML from SkLearn are popping up everywhere

What is next?

Now we have data. So much data. Too much data. It can get noisy, and overwhelming. We also have computer power, which gets cheaper every day. If you add all those items, with the evolution of automated frameworks for artificial intelligence, one would assume that Data Science as a profession will disappear.

The data science applications in production scale that I have ever seen are specific solutions created on a framework to solve a problem, and that problem only. Even if researchers claim to develop a method that is better than the previous ones, when you apply to a different context, with different data, and a different timeframe, the solution will no longer be the best one. That is why automation of artificial intelligence it is a great start, but not a solution to everything.

Data science as we know it will disappear. Companies which don’t have data science as its core will no longer have data science departments. It is extremely costly to maintain a whole department dedicated to it, and usually the results aren’t great. There is a large transformation barrier in companies that are not data-centered to adopt to the ever-changing environment of data science. I will not even discuss the difficulty in setting up the development itself - have you ever tried to install Python on your office computer? And after that, install a package over the proxy that you don’t have the rights for?

I believe that the future of data science as a profession will be more towards companies that develop data products that will run in the cloud and sold as SaaS. That way each company doesn’t need to worry about infrastructure since the solution will be accessed online from everywhere. There is no single solution that fits all, and the future lies in creating small specific AI frameworks to solve specific problems for each department.

I imagine that service provider as being a consulting firm that has people with general knowledge of the companies’ processes and can understand the problem to apply the tools in the best way possible, create a solution and make it available without the infrastructure cost of keeping everything in house. I think that is an evolution of what is called ‘open innovation’. Companies cannot capture all the available talent in the world, and that is just a fact.

The business model for that hypothetical consulting firm is not to leverage hourly workers, where you get bigger by having more people working with you. Instead the value added will be through delivering great data-products that are specific for a problem but can be applied with little to no change to a similar problem in a different company, given that the knowledge base for that solution is already in-house.

As for the future of artificial intelligence, we have seen an evolution from expert advisors to artificial intelligence. I believe that the next step is augmented machine learning, because let’s be honest, a computer won’t ever be able to address complex context and make synapses as fast as a human. However, couldn’t human performance be increased with the continuous support, online learning and immediate feedback from an artificial intelligence? That is another post in itself.

You can also read here, and get more content like this here.