This survey highlights the complexity of language and the challenge of developing AI algorithms capable of understanding and generating language. Over the past two decades, language modelling has evolved from statistical models to neural models, with recent advances in pre-trained language models (PLMs) that use Transformer models and large-scale corpora for improved language understanding and generation.
