Facebook's AI at Work Produces ConViT

Facebook AI says its developed a new computer vision model called ConViT, which combines two widely used AI architectures — convolutional neural networks (CNNs) and Transformer-based models — in order to overcome some important limitations of each approach on its own.?

‘ConViT’, A Computer Vision Model That Improves Vision Transformers (ViT) With Soft Convolutional Inductive Biases

By leveraging both techniques, this vision Transformer-based model can outperform existing architectures, especially in the low data regime, while achieving similar performance in the large data setting.

While AI is still very biased, AI Researchers have been utilizing certain assumptions (inductive biases) that help to train machine learning models.

You can read their July, 2021 paper here.

• CNNs, which have proved extremely successful for vision tasks, rely on two of these inductive biases built into the architecture itself: that pixels near one another are related (locality) and that different portions of an image should be processed identically regardless of their absolute location (weight sharing).

Facebook's ConViT could be important in how Vision Transformers (ViTs) rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification.

ConViT is a New Computer Vision Model

The Facebook AI Research team has developed a new computer vision model called?ConViT. The ConViT system combines two widely used architectures to overcome some important limitations of each approach on its own, namely convolutional neural networks (CNNs) and Transformer-based models.?

The resulting convolutional-like ViT architecture, ConViT, outperforms the DeiT on ImageNet, while offering a much improved sample efficiency.?

The goal of ConViT was to modify vision Transformers to encourage their networks act convolutionally. They introduced a soft inductive bias that allows the network model itself whether it wants to remain convolutional or not.

They did so by introducing gated positional self-attention (GPSA), where the model learns parameters that control how much standard content-based attention is used compared with an initialized position-based one.

Github: https://github.com/facebookresearch/convit

Paper: https://arxiv.org/pdf/2103.10697.pdf

While the future of Facebook is likely the Facebook Metaverse, Facebook AI will have to get moving if its wants to challenge ByteDance (Beijing's alternative to Facebook) in the future of how the metaverse actually works.

While the current AI is hopelessly biased for various reasons, the researchers hope that their ConViT approach will encourage the community to explore other ways of moving from hard inductive biases to soft inductive ones.

The success of deep learning over the last decade has largely been fueled by models with strong inductive biases, allowing efficient training across domains. What will the deep learning of the future bring?

Facebook is Hyping the Metaverse

Facebook has now successful demonstrated bridging the gap between CNNs and Transformers, by presenting a new method to “softly” introduce a convolutional inductive bias into the ViT.

The Metaverse thanks you for your attention. Zuckerberg (who now holds more centralized power at Facebook since the pandemic) envisions a metaverse as an all-encompassing online world where people can game, work and communicate in a virtual environment, often using VR headsets that has up until now largely been depicted in science fiction movies, not reality.