Neural Networks Hate Text

An introduction to word embeddings, one of the fundamental ideas behind generative AI models.

The Internet is mainly text.

For centuries, we’ve captured most of our knowledge using words, but there’s one problem:

Neural networks hate text.

Judging by how good language models are today, this might not be obvious, but turning words into numbers is more complex than you think.

The most straightforward approach is to use consecutive values to represent …