Transformer Models in Natural Language Processing

Abstract

Transformer architectures have reshaped the field of natural language processing by replacing recurrence-based sequence modelling with parallelisable self-attention mechanisms that capture long-range dependencies more effectively. This study presents a controlled empirical investigation of five transformer variants -- the original encoder-decoder model, encoder-only (BERT-style), decoder-only (GPT-style), sparse attention, and linear attention - across six standard NLP benchmarks: sentiment analysis (SST-2), natural language inference (MNLI), question answering (SQuAD 2.0), named entity recognition (CoNLL-2003), machine translation (WMT-14 En-De), and abstractive summarisation (CNN/DailyMail). A total of 1,800 experiments were conducted on identical hardware with a fixed hyperparameter search budget to isolate architectural effects from tuning variance. Decoder-only models achieved the highest performance on generative tasks (BLEU = 28.4 +- 0.6 on WMT-14; ROUGE-L = 41.7 +- 0.8 on CNN/DailyMail) while encoder-only models dominated discriminative benchmarks (F1 = 93.2 +- 0.4 on SST-2; F1 = 92.8 +- 0.3 on CoNLL-2003). Sparse attention variants reduced memory consumption by 58% on long documents (> 2048 tokens) with less than 1.2% accuracy loss, whereas linear attention models achieved comparable speed gains but exhibited more substantial quality degradation on tasks requiring precise token-level alignment. Pre-training corpus size was the strongest predictor of downstream task performance (partial R2 = 0.62), followed by model parameter count (partial R2 = 0.24). A practical selection guide mapping NLP task characteristics to recommended transformer configurations is provided.