On Simulated Annealing
We shall not cease from exploration
And the end of all our exploring
Will be to arrive where we started
And know the place for the first time.
— T.S. Eliot, Little Gidding
The entire study of of Machine Learning could be described as trying to find the tallest peak in a vast mountain range, but without the benefit of an existing map, and without the sense of sight. Let’s say you were dropped off by a helicopter in the middle of the mountains, and are tasked with finding that highest peak.
What would you do?
Maybe you’d start walking a random direction. Maybe you’d eventually stop when you get tired, and say “this is good enough” and call it a day. That strategy is called a “Random Walk”.
Or maybe, instead of walking randomly, you’d try to find what feels like “uphill” from wherever you are, and only head in that direction. That’d be called “Hill Climbing”.
There’s lots more alternative strategies, some which are geared towards parallel computing, and some which are geared towards quantum computing. But they all suffer from the same shortfall.
How do you know when you’ve found the highest peak?
You may have kept heading uphill until there was no more uphill, but is that really the “global maximum” of the mountain range?
Or, more likely, is it just one of many “local maxima”, like how K2 is the second-highest mountain on Earth. Imagine climbing K2, without the sense of sight, and getting to the top, and declaring “This is the top of the World!” only to be told later that sorry, you were only at the second-top.
When we’re super success-oriented, we often worry about this possibility. We look at our choices in our DAG and we worry — what if I end up on K2, instead of Everest?
This is why so many religions and worldviews and modern psychology and pretty much everything except Western Culture tries to instill in us that “it’s about the journey, not the destination”. Because in Life, we’re all climbing hills and exploring possibilities without any map, and without any way to know we’ll find the top.
There’s one last algorithm which I’ve always found poetic. It’s called Simulated Annealing. The term is borrowed from the Japanese art of crafting katanas, where they heat and cool the sword over and over until they end up with something extremely strong — nearly unbreakable.
Imagine that you are dropped off in the mountains, and start climbing uphill, but this time, the helicopter comes and picks you up after a few days, and drops you off again at a different part of the mountain range. As you learn more, and explore more, you can sometimes return to some places and pick up where you left off. Or you can sometimes choose new places to explore, just to see what’s around that riverbend.
The reality of Life is that you can never be sure you've found the “global maximum”. Every “best life” you create and build and sweat and suffer for, may just be one of many “local maxima”. You may realize one day that you’ve “only” reached K2.
But so what?
If it’s mathematically impossible to know when we’ve reach the highest peak, why bother? Why make that an ultimate goal? Why not instead look for a way to live Life where we find the places and the people and the things that make us happy — but then shake things up every once in a while to try something new? Not to diminish those we love, but to get the most satisfaction out of Life?
We’ve all got our own mountain ranges to explore. Each ridge, and peak, and valley has something unique and interesting to experience. Who cares about perfection when the whole point is to enjoy the hike?