Attention-Based Transformer Models for Real-Time Natural Language Understanding at the Edge

Abstract

Transformer encoders such as BERT now underpin most modern natural language understanding, yet their quadratic
self-attention and hundred-million-parameter footprints sit uneasily with the latency, memory, and energy budgets of phones,
microcontrollers, and other edge hardware that real-time voice assistants, on-device keyboards, and embedded sensing systems
depend on. This article synthesizes the literature on making attention-based transformers deployable for real-time edge
natural language understanding, organized around four complementary levers: architectural compression and knowledge
distillation, from DistilBERT through TinyBERT, MobileBERT, ALBERT, and SqueezeBERT; quantization, spanning
straightforward 8-bit schemes through Hessian-guided and outlier-aware low-bit methods and recent post-training
approaches for large generative transformers; input-adaptive computation, including progressive word-vector elimination and
early-exit inference; and hardware-software co-design exemplified by dedicated edge accelerators and standardized mobile
inference benchmarks. Read together, the evidence supports real-time transformer-based edge natural language
understanding as a mature, multiply-replicated engineering pattern for short-utterance classification and extraction on
phone-class hardware, achieved by stacking these techniques rather than relying on any single one, while leaving genuinely
deep, microcontroller-class deployment, longer-context understanding, and rigorous comparison against emerging small
generative models as an open, only partially evidenced research agenda.