NLP Timeline
- Rule-based Era In the early stages, handcrafted rules were defined by analyzing the structure of language. In 1954, the Georgetown-IBM experiment converted sixty Russian sentences into English. These models were unable to preserve context and were complex to scale.
- Statistical Era In this era, probabilistic models are used for modeling sequences, assuming there is an underlying "hidden" state that generates observable outputs. N-gram models and the Hidden Markov model developed in 1970 can predict next word given the sequence of words
- Machine Learning Era In 2001, Yoshio Bengio's proposal of the first neural "language" model using a feed-forward neural network marked a significant milestone. This approach revolutionized NLP by enabling machines to process and understand language in a more nuanced and contextually rich manner. These neural networks, through iterations and advancements in architecture like recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and transformers, significantly enhanced language understanding capabilities. These are used in various NLP tasks such as machine translation, sentiment analysis, and language generation.
- Embeddings Era The Embeddings Era in NLP, particularly the introduction of word embeddings like Word2Vec, GloVe, and fastText, brought about a fundamental transformation. Embeddings represented words as high-dimensional vectors, capturing semantic relationships and contextual similarities between words.
- Contextual Embeddings Further advancing from static word embeddings, contextual embeddings like GPT models have emerged. These models capture context more effectively by considering the entire sentence rather than just individual words.These models use embeddings as a pre-trained representation, decreasing the need for extensive labeling.