A Simple Voice Controlled AI Assistant in C#

TLDR;

A week ago I wanted to see if I could make calls to AI endpoints from inside an XR experience using my natural voice and have the response return in a natural voice as well and managed it successfully. I've since abstracted it into a more generic way so that it can be used in a bog standard WPF or C# application.

The (basic) Architecture

1. Firstly, it records a question or command and saves it to a byte array.
2. Then the audio data is sent to the OpenAI transcription service API to determine what was said and return it as a string (the transcription).
3. Assuming a question is asked, this transcription is sent to the OpenAI GPT API and it returns an answer to the question.
4. Now that we have a response and we want it spoken to us in a natural sounding voice, the text we want to be spoken is sent to the Azure Cognitive Services API (Text-To-Speech API).
5. The audio data is returned back to the original app and played to the user.

Video Demonstration

I've created a video demonstration of the technologies on my YouTube channel and embedded it below.

Azure Text To Speech Voice Gallery

I have to mention that you can have the TTS (Text-To-Speech) generated audio returned in a number of accents, that you can hear here at the Azure Voice Gallery, which is both fascinating and fun to play with.

https://speech.microsoft.com/portal/voicegallery

Brief Speech Recognition and Speech Synthesis History Lesson

For the longest time (up until .NET 5 I think), you could do Speech Recognition and Speech Synthesis offline and locally on your Windows PC using the System.Speech namespace.

Though it was never moved to .NET Core, probably because it relied on a lot of OS/PC specific globalisation language packs? and posed issues running on other platforms such as Mac & Linux. It also sounded very robotic and had limited default accents. A shame, because the System.Speech namespace was nice to work with.

For this reason, this article describes the use of cloud based, online APIs.

Experiment Yourself

If you are going to try this yourself, you will need an OpenAI API Key and an Azure Cognitive Services API Key.

Unfortunately as the code is propriety to the organisation I contract to, so I don't want to share it here. Though I know for certain that those willing and able are capable of getting ChatGPT to generate 90% of the correct code with the right prompts themselves.

Powerful Applications - Get in Touch!

This type of voice controlled AI assistant functionality can be used in many ways.

1. As a standalone desktop voice controlled assistant or a new voice controlled application.
2. Hooked into existing applications - perhaps you have an existing application that you want to add voice control to?
3. Extended so that the intent of your command is understood and various operations execute accordingly?

If you want to explore adding AI voice control & responses to an existing application or the development of an AI voice controlled application, get in touch and we can discuss your thoughts and requirements.

-- Lee

This article was featured in the 2025 C# Advent Calendar. Check it out!