Geometric Deep Learning (a "5G" that might actually be relevant innovation)

One very attractive feature of deep neural networks is what's called the "universal approximation property": a deep and wide enough network can "learn" anything, regardless of how complex, given some fairly lax regularity conditions about the thing to learn are met. That was, in part, behind the strong confidence that many researches had in them, even back when it was even harder than now to train them. Unfortunately, as the field is maturing it is becoming more and more clear that that property might not be so relevant in practical applications, even given the humongous amounts of computing power that we use nowadays. Most of the latest relevant advances in deep learning have been related with new "network architectures", have not relied in "generic" neural networks but have rather used some specific structure that works much better for the problem at hand (be it CNNs for image recognition, LSTM for timeseries analysis, transformers for language models, etc. etc.)

Those structures might seem very ad-hoc, almost like "happy accidents", or in the best case rely on some loose intuitions about the nature of the problem. But using "what we now" about the problem in the structure of the solution is very often the best way to narrow down the complexity and achieve results that might otherwise be impossible using general methods, regardless of how powerful those methods are.

Some people are starting to think systematically about that problem: how do we "incorporate what we know about a learning task in the structure of the deep network to perform that task". And they draw inspiration from an unlikely source: a mathematical "program" from the nineteenth century (in theoretical mathematics, a "program" is a set of related conjectures that are posed together as something worth studying because there might be something "deep" about them... like the current Langlands program or the "sketch of a program" of Groethendieck). Let's make some brief history...

By the mid of the nineteenth century, geometry was becoming much more complex and rich with the emergence of "non-Euclidian" geometries (projective geometry, affine geometry, geometries in higher dimensions, geometries over manifolds, etc.), and there was a sense of unease about it. After all, the view of mathematics as a form of "universal truth" was still very much present at the time, and having different "versions" of something as fundamental as geometry did not fit too well with that ideal. By the second half of the century Felix Klein took the task of trying to unify all those geometries (the "Erlangen program"), and did it in a way that we might consider one of the first examples of "modern" mathematics: instead of trying to "go down" (study further properties and details) he tried to "go up" (figure out commonalities and shared structure). If geometry is "the study of the properties of shapes", let's try to figure out which of those properties remain unchanged under certain transformations of the space that contains those shapes. For example: parallelism in a plane is a property that is maintained if you translate, rotate or even "shear" that plane, but not if you project that plane over an sphere. Angles are maintained over translation or rotation, but not over a shear transformation. And so on. Each "transformation of a space that maintains some properties" is a symmetry of that space, and "symmetries" is of course the object of study of group theory. So Klein was able to reduce the study of the relationships between different geometries to the (much simpler) relationships between the groups and invariants that represented those geometries, and that was an extremely fruitful idea for many decades to come (if all this sounds reminiscent of Noether theorem, that's behind much of modern physics... well, there's a good reason for that, of course!)

How can we apply these ideas to the problem of learning more and more complex functions? (Machine learning is nothing but learning to approximate very, very complex functions in ways that are useful). Well, let's think about transformations of the input that leave the output intact, and ways to introduce that knowledge directly into the structure of the neural network, thus saving it the (rather significant) overhead of having to learn those regularities. That is exactly what Geometric Deep Learning is doing, embarking in an "unification program" very much in the same spirit of the Erlangen program. And, funny enough, when you do that you automatically come up with some well known structures, but also others that are proving its worth right now like Graph Neural Networks.

I will not try to explain the specific examples, in part not to make this article too long, and in part because there is a much, much better explanation than I could ever give in this ICLR 2021 Keynote by Michael Bronstein, which I cannot recommend enough. Suffice to say that the five types of structure that Geometric Deep Learning tries to exploit are represented by "Grids, Groups, Graphs, Geodesics, and Gauges" (which is also the title of the multiple author proto-book that presents these ideas), and those form a "5G" that I am much more interested on right now than the one telcos talk about. There is also a series of lectures on the topic that I am greatly enjoying lately (and which are the reason I decided to write this little article) in here.

I might very well be wrong, but nowadays this seems to me as one of the most promising lines of study about deep neural networks. Of course most of the progress and the news will keep on coming from applications, computational advances, etc. But Geometric Deep Learning has made the mathematics behind deep neural networks fun again for me, and they might do the same for other people. Even for that reason alone, it would already be very much worth taking a look.