I have one small goal for this project. I type a prompt into ComfyUI: Hoshino Ai at a bar, speech bubble, saying "はい". I want the bubble to say はい. Right now Anima, the 2B anime DiT I train LoRAs on in anima_lora, fills the bubble with something that looks Japanese from across the room and reads as nothing up close.

I assumed this was a capacity problem: a small model never really learned the script, and you'd have to train the script into it. A month of poking at it says otherwise. Anima already draws real kana. What it never had was a way to be told which kana, and the reason for that starts with a tilde.

It started with a tilde

I wasn't looking for a tokenizer problem. I was auditing why a handful of captions in my training set encoded strangely, and one of the offenders was a copyright tag:

kaguya-sama wa kokurasetai ~tensai-tachi no renai zunousen~

Every word in that tag is plain romaji. The tilde isn't: Anima's T5 tokenizer can't spell ~, so the subtitle goes in with an <unk> on either side of it. That's a small thing for one tag, so I checked what else becomes <unk>, and it stopped being small. ^^^, ^, ☆ and \ all fold into one shared <unk>, which means four different tags arrive at the model as the same conditioning vector. ♡ is <unk>. So is the · that Chinese captions put between the parts of a name. And native Japanese, all of it, any sentence you like, tokenizes to ▁ <unk> </s>.

To see why that last one matters, you need to know that Anima conditions on two text streams at once. The first is Qwen3 hidden states, which carry the content; Qwen reads Japanese natively, so that side is fine. The second is a table of query embeddings indexed by T5 token ids, a stock English SentencePiece vocabulary, and those queries cross-attend into the Qwen states through a 6-block adapter. Qwen supplies what was said; the T5 queries decide what to pull out of it. If the query stream is ▁ <unk> </s>, there is nothing to ask with.

So a Japanese prompt doesn't get a bad encoding. On one of the two streams it gets essentially no encoding. The first probe (August 15) measured how much that costs: encode a Japanese prompt, and compare the adapter's output with what it produces for the all-English version of the same prompt.

Romanization doesn't help, so "anything but <unk>" isn't the bar. But hand the T5 side an English translation while Qwen still reads the Japanese, and the conditioning comes back. The T5 stream has to be semantic; it doesn't have to be English. That was the encouraging part: the broken piece is small and isolated, and there's a working reference sitting right next to it to imitate.

Two reasons I couldn't let it go

A tokenizer gap is easy to shrug off. Prompt in English, move on. Two things kept me from doing that.

The first is training data. A lot of anime-style data is manga pages, and manga pages are full of Japanese: speech bubbles, sound effects, signs. Train a LoRA on them naively and it learns unconditional text spam. You get gibberish lettering on a café wall, a bubble of pseudo-Japanese floating next to a portrait, signature scribbles in the corner. The standard defense is to mask text regions out of the loss, which works, but throws away pixels and teaches the model nothing about the text. And the mask is only needed because the caption has no way to account for the text: anything I write there in Japanese arrives as <unk>. If a caption could carry Japanese, maybe the text could be explained instead of hidden.

The second is a question I couldn't answer. Anima was pretrained on an enormous pile of manga and booru images, and a large share of them have Japanese written in them. This is a model that tells thousands of characters apart by tag alone. It has seen は and い far more often than it has seen any of those characters, drawn with the same few strokes every time. So why can't it write them? I had been filing that under capacity, a 2B model with no room left for a script, but that answer gets worse the longer you look at it. If every Japanese caption reached the model as <unk>, there is another candidate: it saw all those glyphs and was never once told what any of them said.

The vocab pack

The fix that held lives entirely on the encoder side. I took Qwen's CJK tokens and appended them to the T5 embedding table as new rows (ids ≥ 32128), then distilled only those rows so that the adapter's output for a Japanese prompt through the new rows matches its output for the same prompt with an English translation on the T5 side. Every existing row is left alone, so an English prompt tokenizes bit-identically with or without the pack. And because the new rows are Qwen's own BPE pieces, the two streams stay token-aligned over CJK spans.

The table ended up at 69,558 rows: 58,968 CJK pieces, plus a symbol block (6,118 symbol tokens and 4,472 characters T5 can't spell, kaomoji alphabets included) so that ^^^ and ☆ finally stop sharing a vector. It ships as a text-encoder asset rather than a LoRA, and it's on by default in my stack now.

What it bought: 猫耳 behaves like cat ears in same-seed grids, mixed English and Japanese prompts work, and symbols have their own identity. What it didn't buy is rare-kanji character names. You still type hakurei reimu, not 博麗霊夢. I threw a lot at that (bigger corpora, synthetic name registers, adapter LoRAs at every rank, a vocab swap) and none of it moved. The problem was the target, not capacity: the teacher is a frozen adapter reading an English translation, and nothing in it knows how to compose a rare name either.

Back to the training data

With the pack in hand I could go back to the first of those two problems. OCR the bubbles, put the lines into the caption as quoted Japanese, encode that through the pack, and train unmasked. On a 350-image manga shard (September 1):

I read that as "the quoted text explains the pixels, so the model stops spraying them around," and for three days I believed it. Then I ran the controls I should have started with. An arm whose captions only carry a bare japanese text presence tag, no quotes, matched C cell for cell. So did an arm where the pack's rows were replaced with random vectors of the same geometry. What keeps spam away is having any handle in the caption that soaks up "there is text in this image." The content of the quote was inert. A line of dialogue that appears once in a dataset binds to that image, not to its characters.

So the pack let me train on manga without masks, which was worth having, and blind A/Bs on render quality came out as ties. But asking the model to draw a specific word still gave garbage. The garbage turned out to be the interesting part.

The garbage was real kana

Look closely at what base Anima draws in a speech bubble when you ask for Japanese, and it isn't random strokes. I ran two independent OCR readers over a few hundred of the base model's text boxes, one trained on manga sound effects and one general vision-language OCR model, and counted a box only when both agreed on what it said. On manga-panel prompts they agreed on about half the boxes. The strings they agree on are things like 「いい!」 and 「う…」 and 「だはいい」, plus long nonsense that has real particles (は, を, に, が) and kanji in roughly the proportions actual Japanese has. On signs and storefronts the agreement drops to 1%, and for Korean and Chinese prompts it is zero. Those scripts genuinely aren't in there. Bubble Japanese is.

That answers the question from earlier. All that manga and booru data did teach Anima the glyphs. It doesn't garble Japanese because it can't draw them. It garbles it because the query stream never gave it an address for them. Every Japanese caption it saw in pretraining reached the DiT as <unk>, so it learned that bubbles contain some Japanese, and never learned any way to be told which. English works because the address exists: T5 pieces spell the word.

If that's right, I don't need to teach a script. I only need to wake up what's already there.

Waking it up

The experiment is as frozen as I could make it. The DiT is frozen, the adapter is frozen, the text encoder is frozen. The only trainable object is a delta on the vocab pack's new rows. English prompts never touch those rows, so English output is bit-exact by construction, and whatever comes out ships as an ordinary vocab pack.

The first run (September 13) trained 37 rows covering eight hiragana, あいうえおかきく, for 2,500 steps on synthetic renders of single kana, with the usual flow-matching loss. The caption asks for Japanese text reads as "か". and both OCR readers have to agree on the largest text box for it to count. Untrained, that scored 0 of 16. Trained, 8 of 16 across two seeds, six of the eight at seed 0, each one a single clean glyph, and the English control stayed at 24/24. The delta is about 150 KB. No weight anywhere in the network changed, and it writes the kana you ask for.

A day later the table covered all 92 kana, at 34 of 36 on the same ruler:

Getting from eight to ninety-two took a few findings I didn't see coming.

The first is that glyph identity is decided at high noise. Going from 8 to 24 kana at the same per-row exposure dropped accuracy from 50% to 28%, and か, き and く, which had worked, stopped working. To find out where the information was, I used the model as a classifier: noise a held-out render, score it under all 24 captions, and see which caption wins at each σ. Top-1 was 19 of 48 at σ = 0.8 and at chance nearly everywhere else. Which glyph gets drawn is settled early in sampling, not in the detail steps. The default timestep sampling was spending two thirds of training at σ ≤ 0.6, where the rows have nothing to learn. Restricting training to a band around 0.8 is what got singles scaling again.

The second is that there is no shortcut to glyphs you haven't trained. I wanted unseen kana for free, so I tried a glyph-shape encoder that maps a rendering of the character to its row, IDS composition for kanji, contrastive losses, warm starts. Held-out glyphs stayed at essentially zero through all of it. The trained rows are near-orthogonal random directions, not shape coordinates, so the row-to-glyph map is a hash: the DiT memorizes "this direction means か" and nothing about か's shape lives in the direction. Every row needs its own exposure. That makes kanji a budget rather than a research problem, roughly a GPU-day for the jōyō set.

The third is that a row is a unit, not a glyph. Qwen merges many Japanese words into single pieces (します, ありがとう, 明日), and a single row trained on します draws the whole word. The DiT reads an address as "draw this unit," whatever the unit happens to be.

The fourth took the longest, and it's where I am now. A caption with several trained addresses in a row rendered exactly one of them, and not by position: いやいい came out as いい. My first read was that the DiT can't get order out of these rows at all. But the base model renders multi-piece nonsense English like GLORPAX and MIZUKANE in the right order 22 times out of 24. Reading a sequence of addresses is pretrained. It just isn't available to rows that were only ever trained on single-unit canvases. So I trained on 2–4 piece strings, and a static table did pick up order and count: the first glyph drawn matched the caption's first piece 28 times in 48, against 5 in 48 for the last piece. The price was that singles fell from 34 to 5 of 36. The rows had absorbed "text is several units long" along with everything else. What the table trains on becomes what it thinks text is.

Where it is now

A preview pack is out: 503 rows covering kana, common kanji and punctuation, retrained for quoted-text rendering. Prompt it with something like speech bubble, japanese text. … Japanese text reads as "はい". and single kana or very short words render some of the time, depending on seed. Longer words and sentences mostly don't. On my own target prompt, the honest scoreboard is one exact hit in fourteen renders, against zero for the untouched model. Here is that one:

Left is the base model, where the bubble is a scribble and a stray his appears under the panel. Right is the same prompt and seed through the preview pack. It says はい. It also squeezed the picture into a letterboxed panel, which is the current problem in one image: the rows learned the canvas they were trained on along with the glyph, and nothing in a row table separates the two.

The first fix was the data. Every training item is now a scene composite: the base model draws a character scene with an English speech bubble or sign, the English is erased with the bubble's own colour, and the Japanese is drawn into the cleared region in one of 16 manga typefaces, often vertically. Hard quotas on singles, short phrases and sentences keep any one length from becoming the prior, and the sentence passes train on a lower σ band (0.5–0.9), because order and count get decided later than identity. That work also turned up a bug worth passing on: Adam at lr 1e-3 had erased my warm start by step 50, leaving the table at cosine 0.10 to the rows it started from. A 500-step warmup plus a weak anchor to the initial rows fixes it. If you warm-start an embedding table, log its cosine to the source.

The paired loss

Better data doesn't change what the loss asks for. Plain flow matching on a composite scores the row on the whole picture, so whatever the scene happens to contain gets pulled into the row as bias. When this post first went up, the fix for that was an idea in the queue. It is now the loss that single-glyph training runs on.

The row should learn one thing, the glyph it addresses. So every scene gets drawn twice. B is the composite carrying the new glyph, captioned through the trained row. A is its sibling: the same scene, the same erase, the same typeface, size, position and tilt, but with Latin letters in the box, one per glyph (KZ for a two-glyph string, never the target's romaji), under the same caption with only the quoted string swapped. That is text the frozen model already knows how to render. Both images are noised with the same ε and the same σ ∼ U[0.7, 0.9], both go through the same frozen Anima, and the row is trained on the difference between the two flow-matching residuals, with the sibling's branch detached:

Inside the text box ε cancels exactly, and the target that is left is $x_A - x_B$, the clean pixel difference between the two strings. Outside the box the two images are pixel-identical, so the term reduces to $\lVert v_\theta(B) - v_\theta(A) \rVert^2$: draw this scene the way you draw it for text you can already read. The weight $w$ puts 4 on latent cells under the box and 1 everywhere else. About 80% of the plain residual cancels, namely the scene posterior at σ ≈ 0.8, the erase patch and the paste artefacts, all of it shared between the siblings by construction. The gradient enters through B's caption only and lands on the new rows, and nothing it passes through is updated. The price is one extra no_grad forward per step, 0.69× the throughput.

It does what it was built for. On natural scene prompts, a plain flow-matching table had pushed 20 of 64 renders visibly away from the English-reference render of the same prompt and seed, and at one seed half of the が prompts came out as a white canvas with a single large が on it. The paired table did that to 6 of 64, and kept the scene on every が and ガ image. On single glyphs it is at least even. On a 12-row dakuten set at 1,500 steps, starting from a floor of 0 of 24:

The learning rate is part of the recipe. With most of the residual cancelled, the paired loss is five to eight times smaller than the plain one (0.013–0.02 against about 0.10 on the same items), so at equal lr the rows simply move slower. My first comparison ran both losses at 1e-3 and concluded that the paired one was worse. It wasn't: hits tracked how far a row had travelled, not which loss had moved it.

It isn't free, though. With the scene intact, the glyph lands as scene text, on a board, a subtitle bar or a panel corner, next to lines of Japanese pseudo-text. My readers score the largest text box, so hits where both readers agree roughly halved (21 of 64 against 40). That pseudo-text is not something the rows learned. It is the base model free-running under the japanese text tag, the same real-kana garbage from earlier, and plain flow matching had only been hiding it by wiping the scene along with it. Wiping the scene and keeping the pseudo-text turned out to be two ends of one axis, and mixing the two losses just slides along it.

Sentences went the other way. On multi-glyph items plain flow matching beat the paired loss on every ruler that moved: the paired arm holds the scene and then drops the string into pseudo-Japanese at subtitle size. So training is now two steps over one table. Step 1 binds single glyphs under the paired loss, step 2 adds words, punctuation and order with plain flow matching, warm-started with the warmup and anchor from above, and at the end each trained vector is added into its pack row so that what ships is still an ordinary vocab pack.

The full single-glyph table, 373 rows for 53,000 steps, is training under the paired loss as I write this. It is a rerun, because of a second bug worth passing on. For a day and a half every run had been reading the baked preview pack instead of the raw one, since that is where my config pointed, so several "cold start" results were really fine-tunes of rows that had already been trained. Those results are deleted, and every launch now states its pack. Order is still open: the sentence passes hold the singles where they were, and no sentence ruler has moved yet.

The pack, preview included, is at sorryhyun/anima-vocab-pack-cjk.