ChatGPT in the Age of Generative AI

The human ability to communicate through language is a remarkable capability that develops during childhood and evolves over a lifetime. However, machines can of course not grasp the nuances of human language, their understanding, or communication without powerful artificial intelligence (AI) algorithms. The development of machines capable of reading, writing and communicating as humans has long been a challenge for research.

One of the major methods for enhancing the language intelligence of machines involves language modeling (LM). LM is a technical process that seeks to model the probability of word sequences and predict missing or future words (also called tokens). The goal of LM is to enable machines to comprehend and generate human-like language by learning patterns and relationships between words in a given text corpus.

The field of LM has been the subject of extensive research, which can be classified into four main stages of development. The first stage is statistical language models (SLMs), which emerged in the 1990s and are based on statistical learning methods.

The second stage is the development of NLP which is characterized by the use of neural language models (NLMs). It uses neural networks to model the probability of word sequences like recurrent neural networks (RNNs). The pre-trained language models (PLMs) have become a major focus in the third stage of LM development/evolution.

These large-sized PLMs are now referred to as large language models (LLMs). One prominent application of LLMs is ChatGPT, which adapts LLMs from the GPT series for dialogue and demonstrates a unique ability to converse with humans. Thus, the fourth stage of LM development can be seen as the introduction of these LLMs and subsequent exploration of the upper limits of their performance through the development of ever more extensive and more powerful models.

The emergence of LLMs such as ChatGPT has revolutionized the field of NLP and opened up new avenues for research and development in generative AI. This article provides a brief summary of the current research on ChatGPT and its different versions as a black box. Although significant progress has been made, there are still gaps in our understanding of these models and their implications in various areas. While the field continues to progress, it is crucial to keep exploring the various applications of LLMs. Ultimately, this will lead to the development of more robust and useful tools that can be used by developers and end-users.

ChatGPT is a large language model (LLM) created by?OpenAI?that has been trained on a large amount of data (570 GB/ 175 billion parameters). It has revolutionized the field of natural language processing (NLP) and has pushed the boundaries of LLM capabilities. ChatGPT can generate responses in multiple languages and perform a range of tasks including question answering, creative writing, problem-solving across a vast range of disciplines, writing code, etc.

The popularity and reach of ChatGPT with other popular applications are illustrated?in above figure. Graph shows the time it took for different platforms to achieve 100 million global active users.

The rise of LLMs has transformed NLP by providing state-of-the-art results in various language tasks. ChatGPT is one of the excellent examples, as it demonstrated impressive capabilities in generating human-like language, understanding contextual information, and performing a wide range of conversational tasks. In this regard, a range of survey papers have been published. Existing survey papers focus on specific or general aspects of LLMs or ChatGPT. In comparison, these paper aims to give an overview of the latest advancements in LLMs and put the main focus on the development of ChatGPT.

Large Language Models (LLMs)

LLMs like ChatGPT have revolutionized NLP. They excel in tasks like text completion, translation, and question answering. To effectively train LLMs, it is imperative to have access to a substantial amount of data sourced from diverse mediums such as books, articles, and websites. The data is carefully selected and processed to ensure quality and avoid biases.

LLMs are created using advanced neural network architectures, namely transformers. These transformers have greatly improved NLP by effectively processing and understanding the context of words in a sentence. These architectures consist of numerous attention and feed-forward neural network layers.

To prepare for specific tasks, LLMs undergo pretraining where they are exposed to vast amounts of unlabeled text data. This phase helps models to learn how to predict missing words or masked tokens within sentences, which builds their understanding of language. This process is crucial in enabling AI powered assistants to provide users with accurate and useful information.

Tuning is a crucial step in developing LLMs. It involves training the model on a smaller, more specific dataset that is tailored to the target task at hand. Scaling LLMs involves various techniques. One approach is to train them on wider and more diverse datasets. Another method involves increasing the number of parameters in the model, which enables it to recognize intricate patterns. However, this method demands significant computational resources. Parallel computing techniques and specialized hardware like GPUs or TPUs can also aid in computation scaling.

Optimizing parameters and hyperparameters, such as the learning rate, regularization techniques, and loss functions are essential to attain optimal outcomes when training LLMs. Optimizing these parameters can help the model converge to better solutions and improve its overall performance.

Additionally, human interactions are used to provide feedback, evaluate model outputs, and address ethical concerns, including bias, fairness, and misinformation. Identifying and addressing these issues through interactions is vital for the successful development and deployment of LLMs.

Evaluating the efficacy of extensive LLMs is crucial to gauge their proficiency across diverse tasks. The metrics used to evaluate performance varies depending on the task but typically include accuracy, precision, recall, F1 score, and perplexity. Similar to general ML or NLP methods, it is imperative to conduct comprehensive evaluations to comprehend the constraints, prejudices, and potential hazards that come with extensive LLMs.

Experimenter have recently shown a growing interest in developing and improving LLMs using DL architectures. LLMs are trained on large textual datasets to generate coherent, sensible, and natural responses to natural language queries, and are also used in text generation systems for various applications. It is important to note that the list is not comprehensive as it excludes many competitors, such as DeepMind, Amazon, EleutherAI, BigScience, Aleph Alpha, Huawei, Tsinghua, Together, Baidu, and many others.

Text-to-Text Model

Text-to-Text LLMs are a class of DL models that have gained popularity in recent years due to their ability to transform one text into another. These models have shown excellent performance in various NLP tasks such as question answering, dialogue generation, summarization, and language translation, among others. One of the most?well-known and widely used LLMs is ChatGPT. ChatGPT is designed to converse with humans in natural language and can generate responses to various prompts, including code and math operations.

Text-to-Image Model

OpenAI has developed DALL-E2 which is a powerful GAI model that can produce highly realistic images based on text prompts [62]. The model leverages CLIP (Contrastive Language-Image Pre-Training), a neural network trained on diverse (image, text) pairs, to combine concepts, attributes, and styles [63]. CLIP enables the model to predict the most relevant text description from an image. The CLIP Image Embeddings Decoder module is integrated with the previous model to generate possible CLIP image embeddings from provided text captions.

The IMAGEN model is a text-to-image diffusion model comprising a large transformation LM. This model demonstrates that an LLM pre-trained on a text-only corpus can effectively encode text for image synthesis.

Stable Diffusion is a novel text-to-image LLM developed by the Ludwig Maximilian University of Munich. This model stands out from other existing text-to-image LLMs because of its use of a latent diffusion model, which allows for operations in the latent space for image modification. This approach is much faster and more efficient than previous diffusion models that operate in pixel space.

Text to Audio Model

Google developed AudioLM which takes text as input and generates audio as output generating high-quality audio with long-term consistency. This model uses discrete tokens to represent input audio and treats audio generation as an LM task in this representation space. By training on vast amounts of raw audio waveforms, AudioLM can generate natural and coherent continuations based on short prompts.

OpenAI has developed Jukebox, a music generation model that can produce music with singing in the raw audio domain. It is a non-symbolic approach and enables the creation of music that sounds more natural and authentic as it is generated directly as audio. To achieve this, Jukebox uses a hierarchical VQ-VAE (Vector Quantized Variational Autoencoder) architecture that allows audio to be compressed into a discrete space while retaining as much information as possible.

Whisper is another versatile text-to-audio LLM model that can perform several tasks in the field, including multilingual speech recognition, language identification, and translation.

Text to Video Model

In recent years, researchers has shown promising results in generating short videos from text descriptions, such as generating short clips of animated characters performing actions described in text prompts. Researchers combine techniques from computer vision and NLP to create coherent sequences of frames that align with the textual input.

Phenaki is an advanced video generation model developed by Google Research that uses a C-ViViT encoder, a training transformer, and a video generator to generate coherent and diverse videos from textual prompts. Phenaki generates videos from textual input, making it a significant breakthrough in video synthesis. The model is trained on a large dataset of image-text pairs and a smaller dataset of video-text examples, enabling it to produce high-quality videos with a wide range of scenes and styles. Phenaki is capable of generating videos that are several minutes long.

A new framework called VideoLLM is proposed that uses pre-trained LLMs from NLP as input text for video sequence understanding tasks. VideoLLM includes a Modality Encoder and Semantic Translator to convert inputs from various modalities into a unified token sequence. This token sequence is then processed by a decoder-only LLM, providing a unified framework for different video understanding tasks.

Text to 3D Model

Dreamfusion and Magic3D are two popular text-to-3D models used in the gaming industry to generate 3D images. Dreamfusion is a text-to-3D model developed by Google Research that uses a combination of an image-based approach and a language-based approach.

Magic3D, on the other hand, is a text-to-3D model developed by NVIDIA Corporation that uses a two-stage optimization framework to generate high-quality 3D models from the text.

Text to Code Model

?These are innovative AI systems that utilize natural language descriptions to generate functional code. Codex is one of them, developed by OpenAI which is a versatile programming model that can be applied to various programming tasks. The model uses the technique of program synthesis, which involves breaking down complex problems into simpler subproblems and mapping those subproblems to existing code libraries, APIs, or functions.

On the other hand, Alphacode is an advanced AI system specifically designed to generate functional code for complex and unseen problems that require deeper reasoning. It utilizes transformer-based architectures, including a shallow encoder and a deep encoder, to optimize its efficiency.

Both models are capable of generating functional code efficiently and accurately, but Alphacode is particularly helpful for generating code for problems that require deeper reasoning, such as those encountered in research or data analysis.

Image to Text Model

An image-to-text model is a type of computer vision model that aims to generate a natural language description of an image.

DeepMind developed a visual LM called Flamingo that utilizes few-shot learning techniques on various open-ended tasks involving vision and language. Its unique feature is its visually conditioned autoregressive text generation models, which take in a sequence of text tokens along with images and/or videos as inputs and produce text as output.

VisualGPT is an image captioning model developed by OpenAI that builds on the PLM GPT-2 to generate text descriptions for images. It features an innovative encoder-decoder attention mechanism with an unsaturated rectified gating function, which helps to bridge the semantic gap between different modalities.

Other Models

Some models that do not easily fit into the above-mentioned categories. One of these models is Alphatensor which is a groundbreaking model created by DeepMind that uses deep reinforcement learning to discover more efficient algorithms for computations such as matrix multiplication. This model is based on a game called TensorGame, where the agent (Alphatensor) is trained to find tensor decompositions within a finite factor space.

Another noteworthy model created by DeepMind is GATO, designed as a multi-modal, multi-task, multi-embodiment generalist policy. GATO is a single generalist agent that performs various tasks using the same network and weights. With approximately 1.2 billion parameters, the model is trained at the operating point of the model scale that enables real-time control of real-world robots.

Meta AI Speech Brain is a new model that can directly decode language from noninvasive brain recordings. This is a safer and more scalable alternative to traditional techniques that rely on invasive brain-recording techniques. The model uses a combination of electroencephalography and magnetoencephalography to measure neuronal activity and a DL model with contrastive learning to align brain recordings and speech sounds. The model was trained on 150 hours of recordings from 169 volunteers listening to audiobooks.?

Soundify is a video editing system developed by Runway that aims to simplify the process of finding the right sound and matching it to your video. This is achieved by using a high-quality sound effects library and CLIP, a neural network with zeroshot image classification capabilities.?

In recent years, there has been an influx of LLMs that have been developed and published. These models have a diverse range of capabilities, including the ability to generate human motion, perform language translation, and even create presentations. One notable example is ChatBCG, which utilizes ChatGPT as a surrogate model to generate slides based on the input text.

LLMs are computationally intensive, require a lot of memory, and can be expensive and slow to deploy due to their significant memory requirements. Above figure shows the trend of the number of parameters LLMs in comparison to Moore’s law. ?Additionally, model parallelism and distributed training techniques, such as data parallelism, are being investigated to enable the training of larger LLMs. This approach involves splitting the LLM into multiple smaller models that can be trained simultaneously, reducing the overall training time.

Limitations/ Drawbacks

The LLM-powered search engine may have several limitations. Firstly, the model may generate inaccurate or fabricated information with confidence, as it relies on patterns learned from data and may not always produce reliable responses (training data issue, biased data, fine-tuned data). Additionally, when it comes to dealing with numbers, particularly in the context of financial reports, the LLM model may struggle and may not always provide accurate summaries or interpretations of numerical data.

Since its initial release, ChatGPT has undergone significant improvements leading up to the latest version, GPT-4. However, it still possesses inherent limitations, both fundamental and non-fundamental in nature. OpenAI’s recent updates acknowledge these limitations, including occasional generation of incorrect or harmful content and limited knowledge updated only until 2021.

The use of LLMs for commercial purposes has raised concerns regarding ethics, privacy, copyright infringement, and regulatory compliance. ChatGPT as a widely used LLM with over 100 million active users to date, has been banned in several countries due to privacy breaches. For example, ChatGPT is banned in Italy where a data breach involving payment information and user conversations led to its prohibition. Regulatory compliance with the General Data Protection Regulation (GDPR) has been a major issue for OpenAI, as compliance in some countries remains uncertain.

Conclusion

ChatGPT has brought about significant changes in the way humans use and interact with language. Its ability to generate human-like responses in multiple languages has made communication more efficient and accessible in the short term. However, its long-term impact on the evolution of language remains uncertain, with potential risks of language simplification and increased language barriers. Moreover, concerns about bias and fairness arising from ChatGPT’s reliance on data and algorithms need to be addressed to prevent the perpetuation of existing linguistic biases and inequalities.

• Large Language Models (LLMs) can understand complex sentences, understand relationships between entities and user intent, and generate new text that is coherent and grammatically correct.
• The article explores the architecture of some LLMs, including embedding, feedforward, recurrent, and attention layers.
• The article discusses some of the popular LLMs like BERT, BERT, Bloom, and GPT3 and the availability of open-source LLMs.
• Hugging Face APIs can be helpful for users to generate text using LLMs like Bart-large-CNN, Roberta, Bloom, and Bart-large-CNN.
• LLMs are expected to revolutionize certain domains in the job market, communication, and society in the future.

ChatGPT, as an LLM, has continually improved and expanded its capabilities. The future of ChatGPT appears promising, with potential avenues for growth and development, including enhancements in NLP, integration with other technologies, personalization, and expansion into domains like healthcare, education, and finance. Researchers from various fields have actively engaged in ongoing research, examining the utility, concerns, and future directions of this technology.

In recent years, there has been specific interest in large language models (LLMs) like GPT-3, and chatbots like ChatGPT, which can generate natural language text that has very little difference from that written by humans. At the same time, the use of ChatGPT has increased deistically. Biomedical researchers, engineers, and clinicians have shown significant interest and started using it due to its diverse applications, especially in the biomedical field. However, it has been found that ChatGPT sometimes provided incorrect or partly correct information. It?is unable to give the most recent information. Therefore, we urgently advocate a domain-specific next-generation, ChatBot for biomedical engineering and research, providing error-free, more accurate, and updated information. The domain-specific ChatBot can perform diversified functions in biomedical engineering, such as performing innovation in biomedical engineering, designing a medical device, etc. The domain-specific artificial intelligence enabled device will revolutionize biomedical engineering and research if a biomedical domain-specific ChatBot is produced.