Grokking on real astronomical data

Grokking, the delayed jump from memorizing to generalizing, has mostly been shown on clean, synthetic tasks like modular arithmetic. I wanted to know if it survives contact with real, noisy, sparse data. The honest answer took two wrong datasets, one genuine 24,000-step delay, and a 100-seed check to get to.

The question

Grokking was first documented on synthetic algorithmic tasks where a "true" underlying rule is exact by construction. I wanted to test it somewhere that rule has to be recovered from real, imperfect measurements instead. Orbital mechanics is a good testbed because it has direct historical precedent: a handful of noisy planetary observations were once enough for scientists to infer Newton's law of gravitation and correctly extrapolate it to bodies never observed in that data, the moons of Jupiter. I trained a tiny transformer on real measured orbital data (central mass and distance in, orbital period out) and asked whether it could recover Kepler's third law and generalize it the same way, to gravitational systems it never saw in training.

Two wrong datasets first

The first version of this experiment used the 8 planets, all orbiting one central mass, the Sun. It looked like a clean failure: the model fit training data but couldn't generalize to moons of the outer planets. That looked like a story about grokking not surviving real data. It wasn't. The central mass never varied anywhere in that training set, so the model had no way to learn how the law depended on mass at all, a data identifiability problem, not a learning failure.

Fixing that took two rounds. Adding bodies around a few more central masses fixed the mass-range gap but exposed a second confound, a distance-range gap between training and test. The dataset that actually worked needed both fixed: 18 real bodies spanning 5 distinct central masses(the 8 planets and Pluto, Earth's Moon, Mars's two moons, five of Uranus's major moons, and Charon), trained for 60,000 steps.

It generalized, and it grokked

With both confounds resolved, the model correctly generalized Kepler's third law to 12 real, held-out bodiesacross three gravitational systems it never trained on, the moons of Jupiter, Saturn, and Neptune, with errors mostly under 10%. And the run didn't just generalize, it grokked in the textbook sense: training fit converged by roughly step 3,600, but held-out accuracy didn't converge until around step 27,600, a genuine ~24,000-step delay between memorizing and generalizing.

I also checked this wasn't just a coincidence of the output numbers. The model's locally estimated mass and distance exponents on the 12 held-out bodies (−0.448 ± 0.026 and 1.417 ± 0.044) sit close to Kepler's true values (−0.5 and 1.5) and stay tightly clustered across all three unseen systems, so the model recovered something close to the actual physical law, not just numbers that happened to work.

Then I checked if it was luck

A second seed replicated the successful generalization and the exponent finding, but took a noisier, longer, multi-episode route to get there rather than seed 0's clean single delay. Two seeds isn't a robustness claim, so I scaled the check to 100 independent seeds on cloud infrastructure. The 2-seed result turned out to be optimistic: only 29 of 100 seeds (29%) reached successful cross-system generalization within the 60,000-step budget. The other 71% never converged to a generalizing solution in that time.

The part that kept this from being a purely negative result: among the seeds that did succeed, the recovered mechanistic exponents stayed reliably close to the true values (mass −0.43 ± 0.04, distance 1.37 ± 0.05 vs. true −0.5/1.5). So when generalization happens, it reflects real recovered physics rather than a fluke, it just doesn't happen most of the time at this training budget.

What I take from it

Grokking-style delayed generalization does survive contact with real, noisy, non-synthetic data, and when it succeeds, it reflects genuine recovery of the underlying physical law rather than memorization dressed up as generalization. But success itself is seed-dependent and happens in a minority of random initializations at this training budget, and the exact shape and duration of the pre-generalization delay should be treated as noisy, not as a precise, reproducible constant. The two dataset failures before the real result are as much a part of the finding as the 29%: a single-system training set wasn't enough for a real model, the same way it historically wasn't enough for the scientists who first inferred this law.