Intel Day kicks off at AI Field Day 4

It is day 2 of AI Field Day 4, so I'll be sharing my thoughts in real time all day long today. We're starting off the day talking about Xeon CPUs.

Deploy AI Everywhere on Intel Xeon CPUs

Ro Shah, AI Product Director at Intel, started off Intel Day. Intel talks about AI from the entire flow perspective. But today Shah is going to talk about the inference part of that flow.

Their customers are either large cycles dedicated to AU, so clusters based on GPUs and other AI accelerators. Or they deploy general purpose AI, customers trying to infuse AI into existing applications. Intel sees these customers using CPUs for their AI deployments.

Prior to genAI boom, most models were under 1B parameters. The example Shaw gave for a mixed-use environment (general + AI uses) was video collab. These customers can make use of Xeon chip instead of using more expensive dedicated accelerators.

Generative AI is on the hundreds of billions of parameter models, which changes the state of GenAI. But many enterprises are more likely to deploy smaller LLMs (20B parameters). This is where Intel is looking to deploy Xeon.

Next token latency should be less than 100ms. That's faster than a human can read, and that's a baseline customer requirement for customers, and something Xeon can handle (using all cores). The tests results Shah showed were for a single user.

1st token latency (compute bound), next token latency (memory bound). More cores is good for the 1st token, more memory is good for next token. That's on the Intel Xeon roadmap.

Intel's primary goal has been to upstream for developers, so they offer extensions for things like PyTorch and other common tools.

Intel deploys AI everywhere

So how does a chip company like Intel go beyond the hardware? They have good partnerships and a great ecosystem. CPUs aren't THE solution for inferencing, but they can solve a good chunk of use cases customers have.

#AI #AIFD4 #Intel #Xeon