From Vocabulary Lists to Vectors: The Math of Learning a New Language

Living in Sabadell, I already speak the language well enough for daily life, but lately, I have been dedicating a lot of time to learning how to properly write in Catalan. When you learn any language, a very common strategy is to study a list of the most used words. However, a word by itself doesn’t mean much; its true meaning comes from the company it keeps. I wanted to build an algorithm to track how these words connect to each other.

My first mathematical instinct was to use a Markov Chain. The idea is quite simple: if we know the current word, can we calculate the probability of the next one? We could count how many times a word follows another in a book and build a transition matrix.

The Problem of Giant Matrices

Here is the issue. If the book we are analyzing has a vocabulary of unique words, our matrix will have a size of . That is cells!

And the worst part? Most of these cells will be zero because most word combinations never actually happen in real life. In mathematics, we call this a sparse matrix. It takes up a huge amount of computer memory and doesn’t really tell us much about words that share similar meanings but never appear exactly next to each other in a sentence.

The Solution: Word Embeddings

To fix this, we can reduce the size of the problem using something called Word Embeddings. A famous paper that completely changed how we do this is Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al., 2013).

Instead of counting exact word matches and building a giant, empty matrix, this model uses a small neural network to compress the information. It transforms every word into a small, dense vector of numbers (usually with to dimensions).

The Mathematical Vision

In this new vector space, geometry represents meaning. Every word becomes a vector . If two words are used in similar contexts, their vectors will end up pointing in the same direction in space.

To find out if an unknown word is similar to a word we already know, we just need to measure the angle between their vectors. We do this using a standard formula from linear algebra called Cosine Similarity:

Here, and are the vectors of the two words. If the result is close to , the angle is very small, which means the words are used in the same context and probably have almost the same meaning!

Ready-to-Use Libraries

The best part about all this is that you do not need to train these models from zero. There are amazing open-source libraries in Python, like Gensim and spaCy, that already contain pre-trained data for many different languages. You can simply download them, input a new word you just read, and ask the algorithm to find its closest mathematical neighbors. It is a beautiful mix of mathematics and linguistics that makes learning a language much more interesting.