Character-Level Decoder-Only Transformer

Large language models often feel like black boxes. The goal was to implement the central building blocks of a decoder-only transformer in PyTorch, train the model on a small text dataset and evaluate what it actually generates.

A character-level language model with deterministic character tokenisation, learned token and position embeddings, multi-head causal self-attention, RMSNorm and residual blocks with feed-forward layers, where the language-model head shares weights with the token embedding. The model is trained to predict the next character given a prompt and generates text autoregressively.

I implemented the model in PyTorch. Text is tokenised deterministically per character, with explicit character–ID tables that fail clearly on unseen characters. Attention uses a fused QKV projection and a causal mask, inside pre-norm residual blocks. The main configuration had six layers, eight attention heads, model dimension 256 and a 256-character context. Training is masked autoregressive: only the target text, not the prompt, contributes to the loss. It used AdamW with gradient clipping, validation, checkpointing and a fixed seed, on text pairs from a small translation-style text task. Generation supports temperature and top-k sampling as well as beam search, and evaluation uses custom character-level implementations of BLEU and ROUGE-L.

In the final evaluation the model reached BLEU 0.2304 and ROUGE-L 0.4407. Both are custom character-level implementations of the metrics, measured on the project’s small translation-style evaluation, not on standard language-model benchmarks.

The figures are not comparable with word- or token-level BLEU and ROUGE scores or with results from large language models. The model is small and trained on a limited dataset; the point is insight into the architecture and training loop, not a competitive language model.