2025 Funday Friday #10: The Future of LLMs is Song

tl;dr-

• Large language models (LLMs) form the basis for much of the most popular artificial intelligence products today, ranging from ChatGPT to Deepseek
• The 'soul' of LLMs is found within the deep trends in human language - itself a systemized abstraction of the external world created fundamentally from how humans intake, process, and organize information; with language as the main training data set, LLMs are able to create a facsimile of human thought patterns that builds a primary output
• Using this premise - that the 'soul' of LLMs is found within the deep trends inherent in how humans intake, process, and organize information, a future advancement may be found in augmenting current data training sets and methods with a second systemized abstraction independent of human language that also has enough readily available data to support an LLM
• Luckily, such a system exists - it's called music
• While there are multiple LLM projects focused on creating generative music (many of which are potentially 'really good'), one impeding factor - as mentioned here - is a general lack of paired audio-text data to faciliate LLM training; this is problematic as most current LLM approaches seem to requiring pairing 'sounds' with 'text descriptions' (for various methodological reasons)
• Accordingly, a potential breakthrough may be found in the creation of an LLM that can generate music without the need at all for paired audio-text data at all; this potentially will allow the LLM to generate music more like a human, as humans do not innately pair music with text during the experience
• Yet, this would likely require reimagining how we design the reward system / training process for LLMs; a reimagined reward system and/or training process will likely have to be based not on text, but in either a biometric response and/or an action that someone takes based on the generative output from that LLM

How do humans understand information?

Text - written language - has become ubiquitous because it is the most efficient method. It doesn't require major specialist knowledge that other forms of communication may require. It also doesn't demand synchronicity that methods such as live communication need to function. It's easily stored relatively speaking. The more I think about it, the more it makes sense that text has taken the world by storm in a sense as the primary way we understand each other and store those insights for the future.

But conversion from a concept to text is not a lossless compression algorithm. To boil a song, for instance, with all of its history, meaning, and even the contours of the sound itself across an infinite amount of frequencies, into a set of words is to remove a great deal of information that impacts how someone experiences and reacts to music. Often, this reaction to a musical piece is highly contextual on the individual. Accordingly, current approaches to generative music suffer a 'double cut' from data loss - one from the unique way in which someone listening to a piece may describe it in text, and two from the innate loss of information converting anything abstract object into text.

Solving for the second cut above - this idea of text not being a lossless compression of an underlying idea - is philosophically what I believe will enable AI to create more 'human-like' outputs. Conceptually, it's quite simple; by designing how a synthetic model 'understands' information - intake, contextualization, storage - to be as human-like as possible, we are likely to create outputs that consequently, are more human-like than if the model is processing information in a more 'artifical' way. The fact that current LLMs have achieved their level of performance feeding off of language- a very 'human-like' way of communicating - lends credence to this fact.

It also means that at some point, AI will have to feed off of information that isn't in textual form - and I can't think of a better place to start exploring that dynamic than generative music. How music serves as a communication channel from people to people is completely independent of text; it is a process in which auditory soundwaves across different frequencies and patterns elicit a chemical reaction that invokes some kind of biological response, all without 'the spoken word' ever being used. In fact, introducing the dynamic of 'paired text' likely overcomplicated this process because people, frankly, aren't experience the music 'in textual form.' Yet, LLMs currently can't function without paired-text because the way an LLM 'understands' success - the reward function or RLHF, etc. - are all based on textual inputs.

What other measure could you use? Well, I like the one you see in this video here with award-winning genius musician Jon Batiste; you measure what someone does with the music. Do they listen to it again? Do they riff off of it? Do they incorporate it? Do they take a meaningful action - because the action 'fuzzily' implies the music has changed something in them.

So now, instead of having an LLM generate music and be rewarded because it associates the generated piece with the word 'cool' - and that matches a human also associating that piece with the word 'cool,' you change it up. The LLM generates music and becomes rewarded because it's associated that piece with the action of 'listen to it again' - and that matches the human actually listening to it again. Seems quite a bit like designing a feedback loop for a video game, doesn't it?

Take it a step further - we know that music can result in physiological changes in the human brain. Imagine the reward system for an AI you can build when, instead of being limited to text and language, you are training it to understand feedback in a whole other dimension.

Maybe then, you train it to understand information across multiple whole other dimensions without parameterizing it all through text! Because, after all, we humans don't just take in feedback from our eyes or ears, right?

I wonder if that future will be more a Sabrina Carpenter song, or a 'dun dun dun'? Perhaps I may even live to find out.

Talk soon.

-WY