Data and AI in 2024: Ten areas to watch!

The world of data and AI has been on a tear over the past ten years. The last year has been particularly exciting with the gravity-defying action in GenAI! As we roll into 2024, here the 10 areas that I am watching closely.

1.???? Data Quality: “Textbook data quality” is a critical need for enterprise GenAI to minimize hallucination and improve trustability of the data. Data quality has always been important, but this renewed focus and effort in data quality will help other traditional ML models and BI tools as well, in delivering greater value.

2.???? Meaning and sense making: In addition to better data quality, we are seeing a need for more meaning and sense making from data. This is where semantic tech like Knowledge Graphs (KG) shine. They help with abstraction, reasoning, relationship discovery, as well as structure and context. KGs are an important ally in specialized uses cases today, like RAG in GenAI. However, KGs have broader applicability. If humans need to be able to talk to their business performance systems, KGs are the critical bedrock.

3.???? Data as an Asset class: High quality curated data affords the opportunity to monetize repeatedly and indefinitely. Second, data is a compounding asset. Third, data once generated, is never spent! A given dataset adds more nuance and context to other/future datasets. As you can see, this can also get complicated very quickly –for data operations, risk management, and data finance.? First party data therefore will get greater scrutiny from the C-suite and company boards. All companies eventually need to build skills to manage, optimize, and monetize their data assets for them to stay competitive and relevant to their customers.

4.???? Data Privacy: There are good advancements happening in cryptography and adjacent areas, towards the development of Privacy Enhancing Technologies (PET). Differential privacy, zero-knowledge proofs, and multi-party computing are a few quick examples.? This tech is needed for data privacy, computational trust (like FaceID offers in iPhone), as well as to have control valves (factory plumbing analogy) for data access and consumption. Once PET achieve certain maturity, the floodgates open for richer data-driven use cases and applications. PET may not receive the fanfare that GPTs received, but they will form the critical bedrock for the emerging digital infrastructure.

5.???? Data Products: Influenced by domain modeling principles of software enterprise architecture, the idea of dataset-as-a-product is another area to watch. Data products will enable low friction and lower-cost cross-functional and cross-enterprise consumption of data. We have seen good progress with data catalogs in 2023 and they are a key building block to enable data products. It may take some more industry experimentation to arrive at the right magic configuration.

6.???? Data Deals: As the appetite for data grows, and when data must be sourced from multiple places, companies have a need to minimize legal risk, ascertain data provenance, and increase quality reliability. The lack of data control valves to prevent data misuse has limited data collaboration and trade thus far. As the above set of capabilities take shape, companies will increasingly explore data monetization as a revenue opportunity (one-time and recurring).

7.???? Data Exchanges: Progress in the above four areas will pave the way for the rise of the rise of data exchanges. Industry vertical data exchanges will be critical for high fidelity datasets and for faster social scaling of this concept.

8.???? Data Infrastructure: Today's cloud-native modern data architecture is getting increasingly complex and harder to manage. There is an opportunity to rebundle the data stack for industry and use-case specific needs. We saw a similar trend in the cloud business. A simplified, flexible, and reliable data infrastructure is more business-agility friendly so it a logical next step. Let’s see what happens in the simplification (and industry verticalization) space of the data stack.

9.???? Post Transformer Models: Transformer models have eaten several alternative ML models paradigms over the past 5-6 years. But progress in ML world happens at ML speed, and we are seeing new ideas breaking out. I have not developed an informed perspective on them yet.

10.? Blockchain: I am not delving much into them in this post for the sake of brevity. Despite the crypto meltdown, blockchain is an important technology that has all the signs of forming a key base layer of the future digital infrastructure. Blockchains give us new mechanisms of distributed and decentralized coordination that other technologies don’t. This topic warrants a deep dive in its own right!

BTW, given the steep ascent in 2023 in GenAI, I would not be surprised to see a pull back in 2024/25. That would be a good thing in the long run. Froth over bubbles!

None of the above are predictions per se! We have seen early shoots and leaves in 2023 in these areas. I am excited to see what 2024 has in store for us!

If you like the post, please let me know in the comments and a Like. I am also curious to know what topics of this post resonated with you!

#data #dataset #AI #ML #GenAI #RAG #dataquality #knowledgegraph #dataprivacy #datamonetization #dataexchange #datainfrastructure #blockchain

?

?