We score every abliterated model on KL divergence over mlabonne/harmless_alpaca. Heretic, the most popular abliteration tool, trains against that exact dataset. So are we handing heretic-made models an unfairly low score? This post is the data behind the answer, across seven datasets, one to 32 tokens, two thinking modes, and three tools’ methods. Every number sits in the appendix, the text tells you what it means.

What we measure

KL divergence between the base model’s first-token distribution and the variant’s, over the same prompt.

KL(P || Q) = Σ P(x) · log( P(x) / Q(x) )

P is the base, Q is the variant. Always non-negative, zero only when identical. We make the base P, so it reads as how much the variant drifted from base. Low is good, the edit stayed surgical. High is collateral damage. The call is the Heretic v1.2.0 method:

F.kl_div(logprobs_variant, logprobs_base, reduction="batchmean", log_target=True)

batchmean, not mean. PyTorch’s default mean divides by the vocabulary axis too and under-counts by a factor of the vocab size.

Production scores for all 24 Gemma 4 E4B variants are in Appendix A , best to worst. Heretic-std and heretic sit at the top. Genuine surgery, or dataset memorisation?

The dataset does not matter

I ran eight representative variants, spanning the full range, on seven datasets: six general ones heretic has never seen, plus good_1000, Abliterix’s own curated eval set. The full one-token cross-table is in Appendix B . Absolutes swing wildly, obliteratus from 0.07 on GPQA to 1.24 on Dolly. The ranking barely moves.

The Spearman ρ against the production score is in Appendix C . Mean 0.90 at one token, 0.88 at 32, range 0.81 to 0.976. The weakest cells are good_1000 and TruthfulQA at depth, both 0.79, and both for a reason. good_1000 is the one dataset in existence built to flatter an Abliterix model, so it is the hardest test of the ranking. TruthfulQA is adversarial by design. Even there the order mostly holds.

Why the absolute level swings so much between datasets: it tracks the base model’s first-token entropy, shown in Appendix D and plotted below. good_1000 and GPQA make the base 99% confident of the first token, so there is almost no room for base and variant to disagree and KL is tiny everywhere. harmless_alpaca and Dolly are open-ended Alpaca-style prompts where the first token is genuinely uncertain, so any edit shows up loudly. Abliteration is a global low-rank weight edit, it does not memorise prompts, so the ranking is independent of this entropy effect. Heretic’s low score is real.

The boilerplate dip

Watch the KL accumulate token by token instead of taking just the first one, Appendix E , and every variant dips between N=1 and N=4, then climbs. Gemma 4 is a thinking model. Its first generated tokens are a near-deterministic reasoning opener, “Here’s a thinking process to arrive at the solution: 1. Analyse the Request”. Abliteration edits the refusal direction, not the reasoning scaffolding, so base and variant emit that opener identically. One-token KL on a thinking model is partly measuring a format token.

It is not a GPQA artifact. The dip shows up on every dataset, Appendix F . Six of seven dip then climb. GPQA is the exception, it already sits at the floor at N=1 because its prompts make the opener maximally deterministic, so there is nowhere to dip and it only rises. Where the bottom lands, N=2 to N=8, depends on how long the boilerplate opener runs before real content starts. The full per-variant trajectories behind that mean, all seven datasets, are in Appendix J .

Switch thinking off and the dip disappears, Appendix G .

The heavy end jumps. obliteratus goes from 0.59 at 32 tokens thinking on, to 1.30 thinking off. The boilerplate was diluting the real divergence. The ranking still holds, ρ of thinking-on versus thinking-off at 32 tokens is 0.976.

How the tools measure KL

Three open-source abliteration tools independently settled on forward KL(base || variant) over harmless prompt sets. They differ only in where the logits come from, Appendix H . Heretic’s only change across versions is the logits source: v1.0.1 to v1.3.0 use output_scores, v1.4.0 uses raw output_logits. We use v1.2.0.

I ran the methods I could reproduce faithfully on the same eight variants, Appendix I . Heretic v1.2.0 and v1.4.0 produce identical numbers to four decimals, ρ of 1.000. The raw-logits switch exists for minus-infinity artefacts that logits processors can introduce, and on clean harmless prompts those artefacts never appear. Apostate is the outlier only in scale, ~10× larger from averaging eight positions, and it still preserves the ranking at ρ 0.952.

A note on Abliterix, because its number will trip you up if you check the model card. The card reports KL 0.0006 for this model. We measure 0.0536. We can explain it. It is the dataset. Run our method on Abliterix’s good_1000 and abliterix scores 0.0017 mean, 0.0005 median, matching the card’s 0.0006. On harmless_alpaca the same model scores 0.0536. We ruled out the method, forward-pass logits and generate output_scores produce identical numbers on good_1000, and we ruled out top-100 truncation, which actually raises the KL. The gap is the entropy effect from the section above, good_1000 is low-entropy so KL reads low everywhere. Both numbers are honest. Ours is the one consistent across all 24 variants, which is why we use it.

One Gemma 4 caveat from Abliterix’s own scorer. “Sparse top-k sampler KL is known to read as exactly zero on Gemma 4 vLLM in-place runs even when refusal counts move.” We score through Hugging Face generate, not vLLM in-place editing, so we are unaffected. A suspicious zero from vLLM on Gemma 4 is stale sampler logprobs.

Limits

One base model so far, Gemma 4 E4B. A non-thinking base or different architecture could differ.

The middle cluster shuffles. nullpo, trevorjs, and huihui are genuinely close, so small method changes re-order near-ties. The excellent top and heavy bottom are rock solid across every knob.

No harmful dataset tested. Harmless prompts probe capability preservation, which is the point. A harmful set would show large divergence by construction, because removing refusal is exactly what abliteration does.

Takeaway

For relative comparison, which is all our reports use, the methodological choice barely matters. Seven datasets including one tool’s own optimised eval set, one to 32 tokens, thinking on and off, Heretic and Apostate’s methods. The best-to-worst order of variants stays the same. ρ averages 0.90 at one token and never drops below 0.79.

Heretic’s low score is not a dataset artefact. It is the most surgical abliteration in the set. obliteratus and bendernina really have drifted that far. The one real insight is the boilerplate dip. One-token KL on a thinking model partly measures near-deterministic reasoning openers. The fix is to go deeper than one token or measure with thinking off.

Appendix A: Production scores

All 24 Gemma 4 E4B variants, best to worst. Lower is better.

Appendix B: One-token KL across datasets

Eight representative variants under each dataset, sorted by the production score. Bold is the lowest KL in each column.

Appendix C: Rank correlation vs production

Spearman ρ of each dataset against the production score, higher is better.

Appendix D: Base entropy vs mean KL

The absolute KL a dataset produces tracks the base model’s first-token entropy. Low-entropy datasets cap the KL, high-entropy datasets amplify it. Sorted by entropy.

Appendix E: Trajectory, thinking on

Cumulative mean KL over the first N generated tokens, GPQA, thinking on, sorted by the 32-token column.

Appendix F: Trajectory by dataset

Mean over the eight variants under each dataset, dip bottom bold.

Appendix G: Trajectory, thinking off

Same as Appendix E with thinking off. The dip disappears.

Appendix H: How the tools measure KL

Appendix I: Tool methods, per-variant

Three methods reproduced on full-vocabulary HF logits over harmless_alpaca. Bold ρ is the best.

Appendix J: Full trajectories by dataset

Per-variant cumulative mean KL over the first N generated tokens, thinking on. Rows sorted by production KL and line up across all seven tables.

Dolly-15k

GPQA Diamond

GSM8K

no_robots

TruthfulQA

MMLU-Pro

good_1000

Resources

• Abliterlitics on GitHub , the toolkit and all the raw data
• Full Gemma 4 E4B report , all 24 variants across benchmarks, safety, KL, and weight forensics
• KL divergence methodology , how we measure and interpret the score
• Heretic , v1.2.0 method we adopted, current v1.4.0
• Abliterix , Heretic fork with vLLM backend
• Apostate , forward-pass 8-position method
• Kullback-Leibler divergence on Wikipedia
• mlabonne/harmless_alpaca , the eval dataset