The BERT is described in the paper 《BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding》.
The RoBERTa is described in the paper 《RoBERTa: A Robustly Optimized BERT Pretraining Approach》.
Now 3 years past. Are there any pretrained-language-model that surpass them in most of the task? (Under the same or nearby resources)
Speedup without accuracy decreasing is also considered as a better one.
Have a look at https://www.topbots.com/leading-nlp-language-models-2020/ and https://ai.googleblog.com/2020/03/more-efficient-nlp-model-pre-training.html or of your choice
https://towardsdatascience.com/bert-roberta-distilbert-xlnet-which-one-to-use-3d5ab82ba5f8
I would also like to add OpenAI's GPT-2 and GPT-3 systems, which have produced some fabulous results on language related tasks. See: https://openai.com/blog/openai-api/ Cheers, Raoul
- Badges
- Science topic