Six (and a half) intuitions for KL divergence | Hacker News

A company provides an internet connection to a business and uses compression; line management can yield profit.

The company encodes tokens with a Huffman code based on its model q(x) — q(x) is the probability the company assigns to token x.

The true token distribution is p(x), the actual probability of each token x.

Average bit length using the company's code is -∑_x p(x) lg(q(x)); optimal Huffman length is -∑_x p(x) lg(p(x)).

The difference, -∑_x p(x) lg(p(x)/q(x)), equals the Kullback–Leibler (KL) divergence.