This post isn't about how to write code — it's about the flow of forming a hypothesis → weighing it, testing it → adopting it or throwing it away. It comes out of my experience working on LoRA for an anime image model, so it may not be all that relevant to other fields. Take it as you will.

1. So what is research, and can you actually do research with AI?

Hmm. I've been through grad school and worn the title of "researcher," but defining research in a single sentence feels beyond me. What I can do is briefly write down the questions and the process I mostly leaned on in this work. For example:

• "So how can I make this model's LoRA training faster?"
• "Would a recently published diffusion training technique apply here too?"
• "How can I generate images faster?"

Let me make these a bit more concrete.

• "The recently released Flash Attention 4 doesn't run on my GPU, but with a few code changes I can get it running. Will it be faster than the existing Flash Attention 2?"
• "There's a LoRA training method with an MoE structure called HydraLoRA. It's only ever been applied to LLMs — can't it be applied to diffusion models too? Reading the paper, it looks like it should work?"
• "There's a paper proposing a simple way to train an i2i editing LoRA. But these LoRAs are particularly poorly shared in the model ecosystem I use. Couldn't I make one myself?"

These questions have no clear-cut answer. No answer means nobody has tried it yet — but it also means there's no built-in way to judge whether an answer is even correct. I run the process of making a question concrete, answering it, and grading it through the following six-stage loop.

idea
  ↓  what must be true for this to be worth pursuing? (gate)
proposal
  ↓  cheap diagnostics first. Phase 0 kills an expensive Phase 3.
bench plan     (PASS / WEAK / FAIL verdict per Phase)
  ↓
bench run      →  timestamped result.json
  ↓
findings       (especially recording what "didn't work")
  ↓
memory         a one-line lesson
  ↓
adopt  |  experiment (off-default)  |  shelve + reason for closing

The key is that whichever way the result lands, an artifact remains. Success goes into an adopted method; failure goes into findings and a shelving record. Even a failed hypothesis is an asset in why it failed — so the next person (or me, six months from now) doesn't dig down the same path again.

So how did the three questions above turn out?

You'll get a feel for it from the endings: how each of the three concrete questions passed through this loop. (Exactly where and how I slotted Claude into each stage I'll unpack separately in Section 2 — here, just the grades.)

Flash Attention 4 — "Faster than FA2?" → rejected. The conclusion was no; it was actually 4% slower. I first got the non-running kernel working via a community fork + disabling compilation per attention, then measured it through Phase 0 (isolating attention timing) → Phase 1 (backward kernel sweep). Over 2 epochs: FA4 5:15 vs FA2 5:01. A fused custom kernel doesn't automatically beat a well-optimized library kernel, it turns out. I shelved it, but "why it isn't faster" remained as an asset.

HydraLoRA — "An LLM's MoE LoRA on diffusion?" → adopted. It works. But I had to change what the router looks at to pick an expert. At first I just plugged it in as the paper described and got stuck, and how I got unstuck is the thing I most want to talk about in this post, so I'll defer it to Section 2.

i2i editing LoRA — "Can I make it?" → adopted (experiment). I made it. But not as an "edit-anything do-it-all" — by narrowing it down to a sub-problem that's actually solvable. The process of arriving at that narrowing — the story of how the diagnostic told me it wasn't the code but the problem framing that was wrong — I'll also defer to Section 2. There's one unverified caveat still open, though, so I'll note that as well.

One rejection, two adoptions. The point is that whichever way the result lands, an artifact and a lesson remain — and that the verdict was rendered not by gut feeling but by measurement (ms/epoch, gate entropy, relative error).

1-b. Even when measurement passes it, one last grade remains — "so is this a contribution?"

Measurement only tells you whether a hypothesis is correct, not whether it's worth anything.

In the previous section, every verdict was a measurement — ms/epoch, gate entropy, relative error, PASS/WEAK/FAIL. This is the spine of the whole post, and it's a principle I have no intention of giving an inch on. And yet, after running this loop for a long stretch, there comes a moment when measurement gives a clean PASS but something in the back of your mind still nags. "The numbers got better… but who does this actually matter to?"

This is the gate I find easiest to skip, yet I think it's the most decisive. The measurement gate and the contribution gate are different axes. The former is internal and absolute (it closes within my code, within my invariants); the latter is external and relative (it can't even be graded without a comparison against what already exists, the SOTA, someone else's baseline).

So at the end of the six-stage loop I hang one more gate. Right before writing findings and adopting it as a method, it's a dedicated spot to ask, "Even if measurement is a PASS, as a contribution it may still be a FAIL."

findings
  ↓
contribution gate   does this result beat what it's compared against? Isn't it something someone already has?
                    (rank-matched baseline · ablation · portability)
  ↓
adopt  |  experiment (off-default)  |  shelve + reason for closing

Why this gate is real is testified to by two cases at once. FA4 was cleanly rejected at measurement (4% slower). Here, both gates pointed the same way. But in a case like HydraLoRA, where measurement tells you it "came back to life (gate entropy recovered)," you mustn't stop there. Changing one line of router source to make it work and what it actually gives you over plain LoRA or an existing MoE-LoRA are entirely different questions, and the latter can only be graded with a rank-matched baseline and end-task quality metrics. That's exactly why my draft MoE-LoRA variant stopped itself at "borderline — a rank-matched baseline is needed first for the main presentation." Measurement PASS, contribution undecided.

The same honesty you apply to leaving negative results as tombstones, you have to apply to positive results too. Not hiding the distance between "the numbers got better" and "it was a contribution" — that's what this last gate does. If the measurement gate kills expensive mistakes early with Phase 0, the contribution gate kills "self-satisfied PASSes" right before adoption. The larger the scope of the contribution, the heavier the evidence burden — a bugfix is fine with a single unit test, but a new method demands a paper citation · a baseline run · a side-by-side over several seeds · a statement of merge viability, and so on. It means proving not just "it worked" but "it has a comparative advantage."

One-line summary · Measurement grades whether a hypothesis is true, and the contribution gate grades whether that truth is worth anything. The former can pass while the latter is shelved, and writing down that shelving honestly is part of this workflow too.

2. So where does Claude fit into this loop?

In my experience, Claude was far more useful as a research assistant running this loop with me than when used as a "code autocompleter." Let me write it out stage by stage.

(a) Splitting up the question and the failure — working out "why doesn't it work?" together

In Section 1 you saw the question narrow from something vague ("how do I make LoRA training faster?") to something measurable ("is FA4 faster than FA2 in ms/epoch?"). This splitting-up is half of research. Lay out a vague idea and Claude pulls out the hidden premises and "what must hold first for this to be true," and I use this as a sounding board before writing a proposal.

But the splitting doesn't end with one pass at the start. The more useful moment is actually when something breaks during implementation. The HydraLoRA I deferred earlier is exactly that example.

At first I just plugged in MoE LoRA as the paper described, and it didn't work — the gate collapsed and all the experts turned on identically. Instead of covering this up with "well, it just doesn't work," I ran the reasoning from symptom down to mechanism together with Claude.

• Symptom: gate entropy pinned to 1.0 and top-1 gate ≈ 2e-4 — the router can't pick anything.
• Why?: when I worked out what input the router reads, it was a mean-pooled text embedding. But averaging 4096 tokens flattens the signal to nearly a constant (std≈0.008). With no material to choose from, of course it can't choose.
• So what's the alternative?: let the router see a more discriminative signal → RMS-pool the rank-R bottleneck activations and read those.

Changing this one line of router input brought it back to life. The key is that Claude didn't toss me the answer — it's that the two of us quickly ran the decomposition down through "symptom → why → alternative." Having a partner to do this decomposition with me at a stuck point was, for me, the most tangible answer to "can you do research with AI?"

(b) Search — "have I done this before?"

When a new idea pops up, my first question is almost always "have I tried something similar before?" But the answer is scattered across many places, so I delegate the search to a dedicated agent — having it separately dig through old experiment records, live code, and past work sessions ("why did I decide it this way back then?").

Example prompt: "Have I proposed something similar to this before? Dig through the old records." → it brings back guards of the "do not re-propose" variety.

When you delegate the search, you get back conclusions, not file dumps. No need to burn my own context grepping by hand.

(c) Design — writing the proposal

When a hypothesis survives, I write a proposal. Here the proposal isn't a promise but a plan with a decision tree attached. The skeleton of a good proposal looks like this.

• one-paragraph core ("the paper in three tricks")
• why it fits this model cleanly
• architecture (where and how it plugs into the existing code)
• mapping loss/data onto our code
• cost & single-GPU feasibility (VRAM/time breakdown)
• implementation stages (dependency-ordered numbered phases)
• validation plan (what would falsify this idea) ← most important
• decision points to settle + arguments for both sides
• risks / honest caveats
• relationship to existing methods

Rather than telling Claude "write this idea up as a proposal," ask "first lay out the validation plan for this hypothesis — what must be true for it to be worth pursuing?" When the gate comes out first, you can kill the hypothesis before even starting an expensive implementation.

(d) Diagnosis — from Phase 0, the cheapest first

The golden rule I repeat to myself most often: a cheap diagnostic kills an expensive implementation.

Phase 0 — spectral statistics precondition        ✅ DONE
Phase 1 — autoregression dynamics probe           ✅ DONE — WEAK
Phase 2 — resolution generalization smoke test    ✅ DONE — PASS
Phase 3 — integration   [conditional on Phase 2 PASS]
Phase 4 — fine-tuning recipe

Each Phase has a runnable script + actual measurements + a PASS/WEAK/FAIL verdict, and that verdict becomes the gate for the next Phase. Not hiding the fact that Phase 1 came out "WEAK" but writing it down as is — that, I think, is the honesty of this workflow. FA4 earlier was the same. Phase 0 profiling stopped me in advance, before I touched months of kernel tuning.

I try to write diagnostics that need no generation first, where possible. For example, measuring only the cosine of two direction vectors without making a single image. And sometimes a diagnostic tells you not that the implementation is wrong but that the problem framing itself is wrong. The i2i editing LoRA earlier was like that. The machinery (attention equivalence, memory) all passed, but what actually tripped it up was the objective setup. In the default setup where reference image = target, the model can satisfy the loss just by copying the reference verbatim — there was no "edit" to learn in the first place.

The payoff came not from a bigger bench but from narrowing the problem → grayscale→colorize. Set the target to a color illustration and the condition to a synthetically degraded grayscale of it, and condition ≠ target is guaranteed by design. Since the structure (grayscale) carries all the spatial information, the text channel ends up carrying only what grayscale can't — color — and the model is finally forced to learn a real transformation. Giving up the "do-it-all editor," I salvaged one sub-problem that actually solves.

(e) Bench — putting it in a standard envelope

I drop experiment results into a standard-format result file. This envelope packs the git SHA, environment (torch/cuda/gpu), full args, metrics, and an artifact list into one file. Thanks to that, even six months later "which commit and which config did this number come from?" is self-evident. When I hand a bench to Claude, I just fire it off in the background and wait for the result — I don't build polling/watching loops; a notification comes when it's done.

(f) findings — especially writing down what "didn't work"

findings is the artifact I cherish most. What goes in here is mostly negative results or precondition checks. The structure looks like this.

• what was measured
• why it turns out this way (mechanism)
• two robust, separable interpretations
• implications (so what to do / not do)
• caveats / how to reproduce (exact commands included)

Every finding is linked to a bench script and contains the exact reproduction command. This, I think, is the line that separates "feeling" from "result."

(g) memory — a one-line lesson

When an experiment ends, I leave a one-line fact as long-term memory. A note that's automatically recalled in the next session. The below are all real entries.

"Lower val FM-MSE did not correlate with better samples." "This model's x0 is visually complete at σ≈0.45." "GAD cosine stays ~0; verified as a geometric ceiling. Stop tuning the AdaLN head."

Memory has discipline too: I don't write down what the code already records (structure · git history). I write only "what was non-obvious," and I delete it if it turns out to be wrong.

3. Small disciplines for working with Claude

Habits for keeping in step with Claude. Most of them were learned by trial and error.

• Delegate the conclusion, don't haul back file dumps. Questions that require sweeping multiple files go to a search agent. Only the conclusion stays in my context.
• For ambiguous judgments, get a second opinion. Non-obvious suggestions like a sign flip, a hyperparameter direction, or "going back to a spot already abandoned" — catch confirmation bias before acting.
• Gate first, implement later. An idea that can't answer "what must be true?" isn't ready to bench yet.
• Record negative results honorably. findings + one line of memory. A dead hypothesis is the next person's map.
• Fire it off in the background and wait. Training/bench is fire-and-wait. Don't build a watching loop.
• Make it report honestly. When a test fails, make it say it failed, along with the output. Don't let it gloss over with "it worked."

And large-scale tasks — "thoroughly audit this method," "try 5 approaches independently and compare them" — can be unfolded with multi-agent orchestration. But since it burns a lot of tokens, I only turn it on when explicitly requested. The pattern is this: several searchers in parallel → adversarially verify the findings (attempt to refute) → adopt only what survives.

4. The evidence burden is proportional to the scope

The larger the contribution, the heavier the evidence it demands. The tiers I use are roughly these.

This table is the institutionalized form of Section 1-b's contribution gate. Requiring different evidence for "it worked" (Tier 1) versus "it has a comparative advantage" (Tier 2) is itself the device that separates a measurement PASS from a contribution PASS.

5. Closing — the spirit of this workflow

If I compress this research culture into a single sentence, it's this.

Decide by measurement, not plausibility; kill expensive mistakes early with gates; and raise a tombstone even for dead hypotheses. And for what measurement passes too, ask one last time, "so is this a contribution?"

I think Claude is not a tool that brings this spirit in your stead, but a tool that helps you keep it together with you. Try having it delegate the search, set up the gates with you, write the diagnostics first, and record the results — good or bad — honestly. Then the you of six months from now digs the same spot over again that much less.