The main announcements from Microsoft Build 2024 conference

Microsoft Build 2024 conference revealed more than 50 new announcements, most of them around artificial intelligence.

Following are the key announcements:

1. GPT-4o model on Azure

The gpt-4o model is a new and revolutionary model from OpenAI, unveiled on May 13th, which brings significant advantages over the previous gpt-4-turbo model:

• Ability to understand human speech, generate human voice, and receive visual information. In other words, the model can now see, hear and understand speech, speak in a variety of languages, and understand human language. There is no need for additional models for transcription or for generating human voice. One model can do it all. That leads to an exceptionally fast response for “chat with a bot” applications.
• The cost of use is 50% cheaper than gpt-4-turbo ($5 per million tokens in input and $15 per million tokens in output).
• The response speed has been approximately tripled (for non-Latin languages, the speed improvement is even more significant compared to the previous model).
• Savings in tokens and cost when using non-English languages. Hebrew text consumes less than half the number of tokens compared to the gpt-4-turbo model. This is of course reflected in a lower price and faster response speed.
• Improvement in the cognitive abilities of the model in understanding non-English languages compared to gpt-4-turbo.
• The quotas for using the new model are significantly higher than its predecessor. This means that many more tokens can be consumed per minute, resulting in much higher output.

Important to know: As of today, the gpt-4o model outperforms any other model available on the market in every test and every benchmark that examines cognitive abilities.

• Microsoft offers a new consumption method called Global Standard. This method allows for a much higher tokens quota (450K tokens per minute for regular customers and 10 million tokens per minute for EA customers). The quota is achieved by automatically routing each model inference to an available cloud region, utilizing the fast communication between regions. This method is currently available only for the gpt-4o model.
• For those interested in consuming the model in PTU, the minimum amount of PTU for consumption is 50PTU (instead of 100PTU required by the gpt-4-turbo model).

2. AI Studio Enhancements

Azure AI Studio, the main tool in Azure for managing and using language models, is now Generally Available. In addition, its vast model catalog has been expanded with more new models:

• Introducing a new model from the Israeli company AI21, which is the only one built with an innovative LLM architecture called Mamba-Transformer using Model as a Service. The new architecture allows for creating a model with less memory consumption, yet still displaying high capabilities and supporting a large content window (256K tokens).
• More new small models from Microsoft have been added: Phi-3-Small (with 7 billion parameters) and Phi-3-Medium (with 14 billion parameters that can be operated in MaaS). These are models that can be run on relatively "weak" hardware and have very impressive capabilities relative to their size.
• A new tiny model called phi-3-vision has been added, which receives both text and image and outputs text. The model is capable of understanding images, detecting entities in the image, and analyzing the situation in the image. The small size of the model allows it to be run on edge devices as well.
• A model from JAIS for handling content in Arabic at a high level and in English for consumption by tokens (MaaS).
• TimeGEN-1 model from Nixtla designed to handle time-ordered information (Time Series) for making predictions and detecting anomalies in token consumption (MaaS).
• Additional models will soon join the catalog from the Israeli Bria AI, Gretel Labs, NTT Data, Stability AI, and a new model from Cohere called Rerank.
• Microsoft is expanding its partnership with HuggingFace, and a plethora of models from there are expected to join the huge model catalog.
• All capabilities of OpenAI Studio have been added to AI Studio, including handling model quotas.

3. AI on your PC

Microsoft is launching a new hardware standard for PC computers called Copilot+. These are PCs that have NPU chips (Neural Processing Units), which add advanced AI capabilities to the PCs running Windows 11. Here are some AI features that will be available for these computers:

• Windows Copilot Runtime - This is a platform for running AI applications on a Copilot+ standard computer. The platform comes with the new Windows Copilot Library that allows developers to add AI capabilities to their applications running on the PC. The platform includes over 40 models designed to run on Copilot+ computers, RAG integration capability for applications (using semantic vectors and a vector store), built-in support for the Pytorch library, and more.
• Recall - This feature will allow you to find any file, document, or image that has passed through your hands using natural language search. For example, if you are looking for a specific document and you don’t remember if it was in an email or WhatsApp or if it’s actually an image of a document – just activate Recall and type the main subject of the document in natural language. The tool will understand what you want and will run a semantic search on the images you scanned with your phone, on the websites you visited, your emails, and your messages.
• Super Resolution - Do you have old photos? This tool will restore and significantly improve them.
• Cocreator - Want to make changes to an image? Want to create new images? With Cocreator, you can change the style of the image and “play” with the image using a prompt.
• Live Captions - Allows you to translate audio passing through the PC in real-time from one language to another. This includes audio streamed from movies and videos. The product currently supports 40 languages.
• Windows Studio Effects - Designed to improve your appearance in video calls. For example, changing the state of your eyes as if you are looking straight at the camera and creating direct eye contact. Other effects the tool offers include advanced noise reduction, lighting improvement, and more.

4. GitHub Copilot Extensions and Workspaces?

GitHub Copilot is the most popular AI tool for developers worldwide, used by over 50,000 organizations. The tool is receiving two new features that take GitHub Copilot a big step forward: GitHub Copilot Extensions and GitHub Workspaces.

• GitHub Copilot Extensions

Now you can add Extensions to GitHub Copilot to allow bringing information from external systems connected to the application, such as: information from the cloud, information from databases used by the application, information from deployment systems, information from internal organizational containers, and more.

GH Copilot assists the developer in working with these external systems and helps solve problems and malfunctions.

• There are already available extensions for GitHub Copilot from third-party providers: an extension for Azure, DataStax, Docker, MongoDB, Pinecone, Octopus Deploy, and many others.
• The GitHub Copilot Extensions feature is currently in Private preview and will soon become public.

• GitHub Workspaces

About 75% of developers’ time is dedicated to non-coding tasks, such as gathering requirements, writing specifications, planning, and more. For this purpose, a new and important feature called GitHub Workspaces was created. To understand the power of this tool, let’s say a developer needs to handle a specific Change Request.

1. Using Workspace, the developer can receive instructions on which changes need to be made in the various code files to implement the Change Request.
2. Then, the Workspace can plan the execution for the developer - step by step.
3. The Workspace knows how to update the documentation and Readme files.
4. Finally, the Workspace activates the copilot to make the requested changes in the code.

The entire process is under the full control of the developer and can be updated and changed by the developer.

The two new tools added will make the developer’s work even more efficient. Github Copilot does the work for the developer, and the developer essentially only serves as a Supervisor.

5. Enhancements to Microsoft Fabric

The Fabric platform was released to GA only six months ago and is already used today by more than 11,000 organizations. Microsoft presents 2 new major features for the platform:

• RealTime Intelligence

There are quite a few information systems that stream large amounts of real-time information to a central repository. For example: IoT systems, telemetry systems, information security systems, fraud detection systems, and more. This information needs to be analyzed in real-time and immediate action is required if necessary (for example, when credit card fraud occurs or when an intrusion attempt is detected in a security system).

Microsoft presents a solution specifically built for these use cases called RealTime Intelligence (in Preview) that rides on the Fabric analytics platform. RealTime Intelligence combines the capabilities of several tools in Azure into one unified solution: Azure Data Explorer, Data Activator, Stream Analytics, and PowerBI.

The user can easily diagnose anomalies, get a real-time situation reports and dashboards, perform advanced real-time information analysis, receive real-time alerts, and even initiate automatic actions (such as blocking a credit card).

• Fabric Development Kit

Microsoft allows developers and ISVs to integrate their applications into Fabric and exposes interfaces for Frontend and Backend, thereby enriching the platform with additional capabilities, additional integrations, and additional control and monitoring tools. The new Kit is currently in Preview.

Thus, ISVs like Qlik can integrate their BI tools or Confluent can integrate their Kafka-based system - straight into the Fabric platform. Additional manufacturers that will use this capability are SAS, Informatica, Dataiku, Striim, Neo4J, Teradata, and more.

6. Team Copilot

You probably already use Microsoft Teams’ capabilities to transcribe conversations and produce a conversation summary. Microsoft takes it a step further with Team Copilot. It is a tool that not only summarizes a conversation. It also produces action items, assigns tasks to people (based on the conversation), ensures that the conversation is conducted according to the agenda set for it, creates notes for you during the conversation, and more. Team Copilot can even generate an automatic post in the conversation chat in Teams with the important information mentioned in the conversation. In addition, Team Copilot can function as a kind of project manager that follows the execution of tasks while also connected to Microsoft’s Planner.

With Copilot Studio, it is possible to develop Agents that perform actions in organizational systems. The combination of Agents with Team Copilot will allow Team Copilot to perform real actions in organizational systems and become an additional team member that helps you manage the work.

Team Copilot is expected to be released for Preview later this year.