Thinking outside the data lake

One of my partners made a joke about some data lakes that are producing nothing but mosquitoes to annoy the business. It is a silly joke indeed. However, it shows how a lot of data projects we here about are still not delivering results.

Often we see in the press some apocalyptic stories on how Artificial Intelligence (AI) and Big Data will automate everything and will take over the human race. Some very smart guys signed an open letter expressing the concern on creating beneficial AI, showing how it can impact in our society. I am pretty sure that more powerful analytics will come out year after year.

Seems to me that this rise of AI and data, and the entire buzz on it, is affecting the mindset, or even self-esteem, of us, human professionals, in a bad way. And that can influence poor results in data projects. Next to these so powerful and magic capabilities, professionals can have a wrong mindset that technology will do the job and underestimate or misunderstand their roles.

Humans must think outside the data lake!

In the first place, algorithms are already thinking inside of it anyway. And within the scope of the data lake, they will do a better job.

Shyam Sankar, makes an excellent point on his TED talk (this is a must see), that we should pursue the right symbiotic relationship between computation and human creativity. This right cooperation between humans and AI / data will only happen if the professionals dominate the big picture, look to computation as part of the solution not as the solution itself.

Another recent article that contributes to this point was published recently in the Harvard Business Review. No matter how big your data lake is already, the right data to surpass the competition might be outside of it. It can be a small dataset or new lake, but you need to be looking further to detect that.

People need to stand as a protagonist, making the best use of AI and data as part of their business solutions and services. Don't let the powerfulness of technology make you step back on creativity, thinking or even feeling. AI and Big Data must cooperate with humans, not the opposite.