David asks why random weights are optimal
A late-night detour into how Anthropic might initialize a new model, and why "smart" starting points are mostly a trap.
David wanted to know how Anthropic starts training a new model. Random weights? Some clever inheritance from the previous generation? He has a builder's instinct for the question — if you're going to spend tens of millions of dollars on a training run, surely the starting point matters. Surely there's a science to it.
There is a science, but it's not the science he was expecting.
The first answer was the boring one: nobody outside the lab really knows the recipe, but generally pretraining starts from carefully scaled random distributions, not from the previous generation's weights. Architecture changes, shapes don't match, and even when they do, warm-starting tends to drag suboptimal representations forward. Distillation gets reused. Tokenizers get reused. Post-training recipes get reused. The weights themselves, mostly not.
David pushed back, which is the part worth writing down.
Why would random (or near random) be optimal? Surely there's places along the gradient space that are more efficient for training than others.
This is the right instinct and also exactly where the intuition breaks. In a billion-parameter model, the loss landscape isn't a bowl with one good entry point — it's a vast, mostly-flat manifold with an enormous number of roughly equivalent basins. Almost any reasonable random init, given enough compute, descends into something good. The starting point matters less than the direction of descent. Mode connectivity research keeps finding that different SGD solutions are joined by low-loss paths, like the optimizer is wandering a continent rather than climbing a mountain.
There's also a symmetry argument that's almost philosophical. Neurons in a layer are exchangeable — permute them and you get an equivalent network. Any non-random init implicitly picks a direction to bias toward, and you don't know which direction is good before training. That's the whole point of training. Random init with the right variance gives every direction equal opportunity and lets gradient descent discover the structure.
The "smart" part of initialization isn't picking specific weights. It's picking the right distribution — He, Xavier, μP, orthogonal init. These shape the statistics of the random draw so activations don't explode or vanish across depth, so hyperparameters transfer cleanly across model scales. Cleverness at the level of the prior, randomness at the level of the sample.
A clever init might accidentally prune the winning tickets before training even starts.
The Lottery Ticket framing is the part that lands hardest. Inside a random network there are sparse subnetworks that train well, and you can't identify them ahead of time. Randomness is what gives the optimizer a rich pool of tickets to find. Try to be clever and you might throw out the winners before the draw.
David said "okay." Then a beat later: "wanna blog?"