Forced/Guided Learning in Deep Learning

The forced/guided type deep learning techniques have proven their ability in any model that outputs in sequences. For example, such type of language models is used in Encoder-Decoder recurrent neural network architectures for sequence-to-sequence generation problems such as:

1. Machine Translation
2. Caption Generation
3. Text Summarization, and
4. Style transfer, etc.

Such types of models/mechanisms are useful in regression prediction like - time series forecasting

Similarly, it has proven its importance and usefulness in training transformer-based models.

Targeted Application Areas

In the following cases, the forced/guided training strategies are useful (if wisely used with supporting factors).

1. Slow convergence.
2. Model instability.
3. Poor skill/quality. (used in the sense to improve the model's skill and stability.)

So, if you feel that you are also thinking in the same direction, and want to know more about such techniques, then the following tutorials will be useful for you.

Tutorials

Reference

1. Bengio, Samy, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. "Scheduled sampling for sequence prediction with recurrent neural networks." Advances in neural information processing systems 28 (2015).
2. Williams, Ronald J.; Zipser, David (June 1989). "A Learning Algorithm for Continually Running Fully Recurrent Neural Networks". Neural Computation. 1 (2): 270–280. CiteSeerX 10.1.1.52.9724. doi:10.1162/neco.1989.1.2.270. ISSN 0899-7667. S2CID 14711886.
3. Lamb, Alex M; Goyal, Anirudh; Zhang, Ying; Zhang, Saizheng; Courville, Aaron C; Bengio, Yoshua (2016). "Professor Forcing: A New Algorithm for Training Recurrent Networks". Advances in Neural Information Processing Systems. Curran Associates, Inc.
4. T. He, J. Zhang, Z. Zhou, and J. Glass. Quantifying Exposure Bias for Neural Language Generation (2019), arXiv.