heissanjay

BPE in Go

Byte-Pair Encoding from first principles: character tokens, pair counts, merges, and a growing subword vocab.

GitHub

A small, readable Byte-Pair Encoding implementation in Go. The algorithm behind most modern subword tokenizers, without pulling in a framework.

Algorithm (as implemented)#

  1. Seed tokens. Split the corpus into words. Each word becomes a char sequence plus a word-boundary marker (_).
  2. Count adjacent pairs across the corpus.
  3. Merge the most frequent pair into a new token everywhere it shows up.
  4. Repeat for a fixed number of merges (or until no pairs remain).
  5. Rebuild a vocabulary from the resulting tokens.

Toy demo corpus in main.go:

low lower lowest

After a couple of merges you can watch common digraphs (e.g. lo, ow) collapse into single tokens. Same pressure that grows real BPE merges on large corpora, just tiny.

Layout#

FileRole
main.gopair counting, merge loop, vocab update, demo main
NOTES.txtstep-by-step BPE notes I kept while writing the code
Makefilebuild helpers

No external ML deps. Just stdlib (fmt, strings) so the data structures stay obvious.

Why it exists#

Tokenization is easy to treat as a black box (tiktoken, HF tokenizers). Implementing BPE once in a systems language makes merge tables, word boundaries, and vocab growth concrete. Good background for any LM or embedding work.