A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

Recently, everyone's interest in ChatGPT did not happen suddenly. Our team (consisting of 9 different schools and companies) carefully crafted and pre-trained the large model for almost a year and a half, and the overview of the pre-training of the large foundation model has finally been preprinted. The overview includes the history of large model changes, different algorithms, and application scenarios, and provides a deep discussion of visual, language, graph, multimodal, and unified foundation models. We hope to help those interested to better understand the pre-training of large foundation models and the technologies behind ChatGPT. We welcome feedback and collaboration from interested teachers and students. Thank you for your attention.

Arxiv: https://arxiv.org/abs/2302.09419