The band post was about where to train a new embedding row. This one is about what the row is once it has been trained, and why that shape made the second half of the project so much harder than the first.

The problem, stated the way it looked from the outside: by the end of September I had a table of about 1,350 trained rows for Japanese glyphs in Anima, trained cold from the vocab pack, and it read short words well. Every attempt to fine-tune it further lost words. Polishing on its own data lost words. Re-banding lost words. Training on real dialogue lines, which was the whole point, lost words. The rows barely moved in aggregate, a few degrees each, and that alone was enough. For about a week every warm run of record came back worse than the rows it started from, and I could not say why, because I didn't know what I was looking at. So I stopped training and measured the table.

The shape of a trained row

Take the trained rows of one family, the 163 kana say, and subtract their mean. What is left is a set of per-row residuals that are nearly orthogonal to each other and nearly equal in length. The mean itself is long, about 30% of the trained energy, and it points nowhere in particular: not along the T5 table's principal directions, not toward any T5 row. Its nearest neighbours in the T5 vocabulary are number tokens and the <extra_id_*> sentinels. I started calling the mean the stick and the residuals the spikes, because a burr on a stick is what the picture looks like, and the names stuck through forty reports.

The full decomposition, read on the shipped rows:

$$\text{row}_i = m_{\text{pack}} + s + (1-\alpha_i)\,q_i + e_i$$

Each term does one thing, and I could check which by taking it out and rendering. Remove $m_{\text{pack}}$ and the composition changes and the text breaks on 7 of 7 strings: the model needs the row to look like a T5 embedding in the one direction all T5 embeddings share. The pack spike $q_i$ carries meaning: on 68 pairs of kanji that mean similar things (左右, 話語), the pack spikes agree at cos +0.21, and on 46 pairs that look alike (日目, 土士) at +0.09. The trained spike $e_i$ is the mirror image, +0.08 on meaning pairs and +0.21 on look-alikes, and its bitmap similarity to the glyph's rendering ranks the right glyph first among 165 kana. Training shrank every $q_i$ by about a third, uniformly, on and off T5's principal directions alike, and left the rest. Swap one row's $q_i$ for another's and the render breaks; project it out and the layout holds with some glyphs changed. The model uses it, but what it uses it for is not the glyph.

The glyph is in $e_i$, and $e_i$ is a hash. I said that in the first post about this project on much less evidence; now it is measured. The spikes are near-orthogonal (the 95th percentile of pairwise |cos| is 0.15, participation ratio 70–80 on 166 rows), near-even in length, and nothing about the shape of か lives in the direction that means か. There is no low-rank layout either: the top principal component of the spikes holds 3–5% of their energy.

One more property that turned out to matter: the whole row sits on a shell. The norm of the full row has a coefficient of variation of 0.05, tighter than the offset alone (0.10), tighter than the T5 table itself (0.17), and it is made by compensation: a longer init gets a more negative offset along it, with correlation −0.82. Training put the rows at one radius from the origin, whatever they started from.

What the stick does

The spikes carry identity, so the obvious guess is that the stick is a bias term you could do without. Two experiments say otherwise. First, scale it: multiply the stick by a factor and leave the spikes alone, on 92 renders of short words from the banner ruler.

At half length no word reads exactly and a quarter survive within one edit. The text region and the slot count stay down to 0.5, and the glyphs inside slide to neighbours or repeat (こんにこちは, アアイイドル); at 0 the base model's own garble comes back. The spikes alone do not read as glyphs. The stick sets where the adapter reads the spikes.

Second, swap it. I had two tables trained cold on the same 81 hiragana: A, trained on scene composites with large lone glyphs, which read words best of anything I had, and B, trained on flat 3×3 grids of small glyphs at low σ, which read words badly and repeated. Their sticks sit at cos 0.875 to each other. Take B's spikes and put them on A's stick, no training, and render:

The grid table's spikes were fine. What it lacked was the stick, and the swap is symmetric: A's spikes on B's stick fall to B's level. Words read on the stick; repeats follow the stick. Here is what that looks like on one word, かなしい, over two prompts:

The stick alone writes pseudo-Japanese; the spikes alone write pseudo-English (Sione ywildefe), which is what the adapter makes of a T5-shaped vector with no stick. The swap that reads best, B's spikes on A's stick, also drops the bedroom on the second prompt and writes the word large on a flat pink canvas. That is the grid table's training picture, a glyph on a blank card, carried in its spikes and released once a stick lets them read. The scene lives in both terms; the swap pairs the worse half of each.

What I could not do with the stick is use it as a knob. ±10% on its length moves nothing. Re-fitting it with the spikes held moves it a long way (cos 0.95 to where it started) and renders the same pictures with the same slot counts. On table A, the same re-fit loses the word fit instead (33 to 19 exact of 104), and the move is orthogonal to the direction that separates A's stick from the sticks of the tables that read worse. The stick has directions the render ignores and one direction the words depend on, and distance between sticks says nothing about which you moved along. Geometry does not call the render.

The stick also isn't the glyphs'. Cold runs on different data rebuild the same stick, cos 0.89 to 0.99 across every kana run of record. The 1,185 kanji were trained in four disjoint cold batches and the four sticks sit at cos 0.96 to 0.99 to each other. Kana and kanji sticks are at 0.75 to 0.80. The stick is the data recipe's: it is what every draw of the recipe teaches every row, and the gradients of two different rows on one item share cos 0.15 at every σ, so every draw feeds it.

Why warm passes lost words

Here is the run that made it concrete. Take the shipped kana rows, hold the stick fixed, and train the spikes on 70% real dialogue lines, 40 steps per row at lr 1e-3. Each spike turned to cos 0.77 of its start at the same length. On the dialogue ruler the short strings fell from 8 exact to 2 of 32 and nothing else moved. Lower the lr to 2e-4: the spikes stay at cos 0.978, and the short strings fall from 8 to 2 again, the same strings. The loss didn't need the turn, the loss at the low lr was no worse, and the cost didn't scale with the turn.

So I turned the spikes by the same amount in a random direction, no training, same stick, same lengths.

The trained direction reads as a random one. Any 12° turn of the spikes costs about five of the eight short strings; the dialogue data's direction buys nothing on the text over chance. That is the finding under all the warm failures: the shipped spikes sit where any turn of this size costs short words, and the gradient the warm runs were following carried no reading signal at that size.

Why not? I held the rows at their start, ran 900 of the training batches through, and read the gradient on each row per draw without taking a step. Split the draws in half at random and compare the summed gradients:

Spearman-Brown puts the signal share of one draw at 2.5%, the same for every group. A row's summed gradient is half signal at about 40 draws; before that its direction is a random walk, and under AdamW a kanji drawn six times in 900 steps has turned before its direction is known. Replaying AdamW against SGD on the saved gradients put them at cos 0.78–0.96: the optimiser scales what the gradient carries, signal and noise alike, and is not where the noise comes from. Neither loss term pushes the norm; the out-of-box term is 2% of the in-box term and inert on the rows. The only structured thing in the gradient was a conflict: for kana, the 2–6 glyph windows and the full dialogue lines pull in opposite directions, cos −0.18 between their summed gradients per row. One row, two layouts, and the data asks for both.

Three more things went wrong that I'd pass on to anyone warm-starting an embedding table.

The norm guard walked the rare rows home. The trainer had a small pull on every free row's residual, there since the cold runs as a guard against norm inflation. Under AdamW a row absent from the batch long enough for its second-moment estimate to cool takes that pull alone, and once the data's trace is gone it steps about 0.8 lr toward zero every step. The frequent kana, in nearly every batch, never saw it. The rare kanji did: after 54,000 steps they ended at cos 0.12 to their start and 7% of their length, the rarest quartile at 0.8%. Five ruler strings by eye: a kanji name gone, 妹属性 drawn as 腿の絆る惟. With the pull off the same run holds every kanji row at cos 0.95 and nearest its own start.

Holding the mean pushes the rare rows around. The ball runs re-centred the rows after every step so the stick stayed fixed. That means every step the frequent rows take is subtracted from every row, and the 30 rarest rows' updates ended at cos 0.20 to minus the sum of the 30 most frequent rows'. The rare rows were being trained on the negative of what the common ones learned.

Freeing everything was the fix, and it was a small one. With both stick and spikes free at lr 2e-4, the spikes went where the stick-held run put them (cos 0.92 between the two updates) plus an 8° drift of the stick, and that drift won back most of what the spike turn had cost: the run ties the shipped rows on every text count instead of losing the short bin. The stick alone, spikes held, runs six times as far in the same direction and loses the long strings. Of the three ways to split the update, the one that doesn't split it is the one that works.

No identity subspace

The last thing I tried on the geometry was the ambitious one: find the subspace of row space the glyph lives in, so that updates could be projected onto it and the page left alone. Two constructions, both borrowed, both stopped on their own rules.

The first fits the second moment of the model's sensitivity to a row, $M = \mathbb{E}[J^\top J]$, from random output probes, separately for probes inside the text box and outside it; the ratio ranks row directions by text-against-page. The lens exists in the sense that the top 16 directions hold a quarter to two thirds of the trace, but it is not estimable at any budget I could afford: two fits on disjoint items overlap at 0.2–0.5, and the tail of one-glyph items dominates. What it did read cleanly is where rows act by σ: at σ 0.5–0.7 a row moves the text box as much as the page or more, at 0.95 the page 10–16× more than the box.

The second isolates identity directly. Draw one item twice, the same scene, font and layout, with one glyph swapped, and take the difference of the row's gradients under the same probes. That difference is the part of the row's effect that depends on which glyph is drawn. Its second moment should be the identity subspace. It isn't one. The two fits overlap at 0.25, flat from 10 twins to 100; the directions where a row's effect turns with the glyph are the same directions where it acts at all, in the same proportion; and the swapped glyph's own row turns no more than its neighbours' rows do. A third of a row's in-box Jacobian rotates when the glyph changes, and the rows for the other glyphs in the window rotate 0.6 to 0.9 as much.

That last number explains the first post's puzzle from the gradient side. At a cold start, swapping the glyph in slot k changed every row of the window as much as slot k's own, and I had hoped that was a cold-start artefact. On trained rows it is the same: what the gradient says about the glyph drawn in a slot is not addressed to that slot's row. Identity is a property the window pays to all of its rows, which is why a loss over the whole text box can't be made to teach one row one glyph, and why a per-glyph credit assignment is on the list for the next version rather than a projection.

What the geometry is good for

Diagnosis, mostly, and one practical thing.

The rows that ship now are the whole-free run above, plus a page-preservation term that is its own post, trained for 40 steps per row at lr 2e-4 with the norm guard off. Against the previous pack it letters less junk and more of the string (glyph F1 0.271 against 0.241, p 0.01), draws the target kanji far more often (recall 0.336 against 0.193), and keeps the page. It still pays on the shortest kana strings, and lines of ten or more glyphs stay at 0 of 32 within two edits for every table I have trained. The pages letter less of what is not the string; they do not letter more of the string.

The practical thing is transplanting. Once the shipped rows were done I needed 225 more kanji, and the decomposition says what a new row has to learn: $m_{\text{pack}}$ and $q_i$ come with its pack row, $s$ is the family's, and only $e_i$ is new. So the 225 were trained in a copy of the table, cold and then one warm pass with every row free, and copied onto the shipped table by value, no retraining there. Their mean sits 12° off the shipped kanji stick, against 6° for a random 225-row subset of the shipped kanji themselves, and the same length to within 2%. Adding the shipped stick's drift to them turned them by under a degree. On the dialogue ruler the merged table and the one it was transplanted onto are indistinguishable on every count. The burr accepts new spikes, as long as they were grown on the same stick.

If I had to compress the week into a rule: a trained embedding table is one shared vector plus a hash, the shared vector decides whether the hash is read at all, and a warm pass over the table is a random rotation of the hash unless each row sees about forty draws of its own glyph. Everything that failed, failed on the last clause.