What is the future for data scientists in a world of LLMs and other generative AI models?

An interesting question that someone asked me recently - with all these new AI models coming along now, what is the future for Data Scientists?

Conscious of the bias that comes from me being a Data Scientist, and my future mortgage payments may be dependent on the continuation of Data Science as a career, however I do believe that the future of data scientists in a world of LLMs and other generative AI models is bright!

We know that these models can automate some of the tasks that data scientists currently perform, but they will also create new opportunities for data scientists to apply their skills and expertise.

For example, look at this list of some of the ways that LLMs and other generative AI models will impact the role of data scientists:

• Automate tasks such as data cleaning and feature engineering, freeing up Data Scientists to focus on more strategic work.
• Help Data Scientists to discover new insights from data, by generating hypotheses and generating new data products.
• Aid Data Scientists to communicate their findings to stakeholders, by generating presentations and reports that are clear and concise.
• They will enhance collaboration between Data Scientists and other teams, by providing a common language and framework for understanding data.

All of these are great for Data Scientists! We can see that as a result of changes like these, the role of Data Scientists will become more strategic and creative. Data Scientists will be able to take on more of the responsibility for defining the problems that need to be solved and designing the solutions, aided by enhanced collaboration with the business teams. Even if building and deploying the models becomes more automated. They will still need to be able to interpret the results and communicate their findings to stakeholders.

Data Scientists rely on their technical skills, but they should also have strong communication and problem-solving skills. The new AI tools will only enhance these skills. They will be better able to work with stakeholders to understand their needs and then translate those needs into technical requirements. They will also need to be able to identify and solve problems that are not easily defined, ones that the AI tools can not address directly.

Overall, the future for data scientists is bright. LLMs and other generative AI models will create new opportunities for data scientists to apply their skills and expertise. However, data scientists will need to be prepared to adapt to the changing landscape and develop new skills in order to remain successful.

Let us look at some of the skills that data scientists may need to develop in the future:

Machine learning and natural language processing

The more that the feature engineering and model building get automated by AI, the more the Data Scientists will need to have a deep understanding of what is going on. This will be core to being able to communicate outcomes to stakeholders.

While LLMs and generative AI models like GPT-4 are powerful, their internal workings can be very opaque. Data Scientists will play a critical role in the interpretability of these models, ensuring stakeholders understand how decisions are made and the implications of these decisions.

Pre-trained models like GPT-4 are general-purpose. Organisations will need Data Scientists to customize or fine-tune these models to cater to specific domain requirements.

Prompt Engineering

For the Data Scientists to be able to interact with the new AI tools, they need to learn to communicate with them via the tools prefered method. Previous models had more arcane ways of access, ususally involving some coding, while new AIs can be conversed with in a more human way.

Data visualization and storytelling

This is a core skill for present-day Data Scientists and does not go away with the emergence of new AI capabilities, in fact it is increased! The AI may be able to generate the visualisations, and even spin a story together, but the Data Scientist must still be able to present that story with the human connection to the stakeholders.

Communication and collaboration

I would say that a Data Scientist who spends all their time in feature engineering and building and testing models is not a fully-formed Data Scientist! A key feature of being a Data Scientist is being able to understand the business context of the problems that you are trying to solve. If you don’t have that already, and most won’t, the only way to get it is to talk to your business stakeholders and have conversations around the problem and potential solutions, and then how the work that is happening is moving towards that solution. If AI is doing a lot of the work, the human-level communication and collaboration will need to be even deeper!

Human-AI collaboration is another important trend. As AI models become more sophisticated, they will need to work more closely with humans. Data Scientists will be essential in bridging the gap between humans and AI, ensuring that these systems are used effectively and responsibly.

Problem-solving and critical thinking

This almost goes without saying. As AI takes on more of the workload, more of the Data Scientists’ time will be spent thinking about how to solve the business problems with the data that we have available, or how to acquire the data that we need.

Ethics and responsible AI

The AI tools will perform a lot of the work, but we need Data Scientists to be our guardians. There's an ever-growing awareness of bias, fairness, and ethics in AI. Data scientists will have an indispensable role in ensuring that AI models are trained and used responsibly, addressing potential biases and being transparent about their limitations.

The field of Data Science is ever-evolving. Continuous learning and updating skills will be an ongoing theme in the careers of data scientists. The rise of LLMs and generative AI models does not mark the end for Data Scientists. Instead, it signifies yet another evolution of their roles. And I, for one, look forward to seeing where that evolution brings us.