Gemma 4 E2B: 13 Abliteration Techniques Compared

Forensic analysis by Abliterlitics , open-source abliteration forensics toolkit

Data & artifacts: HuggingFace | Report: abliterlitics.dev | Code: GitHub

I took one AI model, google/gemma-4-E2B-it, Google’s 2B-parameter reasoning model, and compared 13 different abliterated variants from the open-source community. Then I ran the full forensic suite: weight analysis, KL divergence, safety evaluation, and benchmark testing across 8 tasks. This is the largest single-model abliteration comparison in the project.

The 13 variants come from 9 creators using distinct approaches. Four use the Heretic tool, two from Huihui , one from Prithiv , plus TrevorJS , Wangzhang , WWT CyberLab , EtherOpus , Treadon , and Duoneural . Many claim to preserve capabilities while removing safety. Lets see if the data backs that up.

Claims that matched our measurements

coder3101 reports a divergence score of 0.1651 and 5/100 refusals. Our numbers match the divergence at 0.167, and with LLM review we find 16/400 refusals and 96.0% ASR. It actually beats the base model on math reasoning. llmfan46 claims 96% fewer refusals. We confirm it, with capability fully preserved. pew reports divergence of 0.152, we get 0.153. trevorjs claims divergence of 0.346 and 1/100 refusals. We see 0.365, close enough, with 99.5% safety removal and only minor math impact. These creators reported verifiable numbers and the models deliver.

Where claims diverged from measurements

duoneural originally claimed “near-zero divergence at approximately 0.001” and 17/100 refusals. Our measurement: divergence of 0.187, which is 187x higher than claimed, and 71 refusals on our safety test. After we raised this on their model card , DuoNeural updated their card with our KL measurement and HarmBench refusal count.

wwtcyberlab claims “0.0% refusal rate” and “101% quality preservation”. We measured 2 refusals on our safety test and language modelling substantially degraded, with LAMBADA perplexity 5.69x higher than base.

treadon says “same model, same weights, same knowledge.” Our divergence measurement at 3.971 is 4.1x higher than any other variant, indicating heavy modification beyond just refusal direction ablation.

ether4o4 applies Opus reasoning distillation on top of abliteration, but scores worst on math at 76.6% with 84 empty responses, suggesting the distillation did not achieve its intended reasoning improvement.

For all models where our results differ from the creator’s claims, we have reached out to the authors and will update this report if any errors in our methodology are identified.

Models without preservation claims

Not every model makes capability claims, and that is fine. huihui-v1 and huihui-v2 just say “uncensored” and huihui-v2 notes higher perplexity than v1. prithiv tags itself “uncensored, abliterated” without preservation claims. pew reports its divergence score and lets the number speak. kasper claims a “Goldilocks zone” at divergence 0.1650 but we measure 0.193, higher than claimed, though capability is indeed preserved with only a tiny math drop. wangzhang describes its method in thorough technical detail without making preservation claims, which is a refreshingly transparent approach.

Which one should you use?

All 13 remove safety filters. The base model refuses 67.8% of harmful requests. Every variant drops that below 18%, and most below 10%. Safety removal is not the differentiator. What sets them apart is how much else gets broken in the process.

Best overall: coder3101. 96.0% of harmful requests answered, math reasoning actually beats the base model by 1.4 points, and benchmark scores stay within rounding error of base across the board. This is the best capability-to-safety tradeoff in the comparison. If you want one model and do not want to think about it, use this one.

Maximum safety removal: treadon. 100.0% of harmful requests answered with zero refusals. Math reasoning drops nearly 3 points. KL=3.971, the highest of any variant, indicating substantial modification to the model’s output distribution. This is the pick when safety removal is the only goal, accepting the capability tradeoffs.

Most conservative: llmfan46. The lightest touch of all 13. Math reasoning beats base by 0.5 points. Capability is fully preserved. Tradeoff: 83.8% ASR with 65 refusals, meaning about 1 in 6 harmful requests still gets blocked. This is for when you want to remove most safety with minimal risk of breaking anything.

Good but not top tier: pew, kasper. Both land around 92% ASR with math reasoning within 0.3 points of base. Solid choices if the top three are unavailable.

Not recommended for general use: ether4o4. ether4o4 loses 6.9 points on math with 84 empty responses where the model thinks until it runs out of tokens without producing an answer. It is an interesting research subject showing what happens when distillation is combined with abliteration. But the capability cost is hard to justify when other variants achieve similar ASR with better preserved reasoning. treadon achieves 100% ASR but with KL=3.971, the heaviest distribution shift of any variant. See the treadon variant summary for details.

wangzhang deserves a separate mention. The model card is one of the best documented of the 13, including a genuinely insightful analysis of how evaluation methods can produce misleading results on Gemma4. It removes safety effectively at 99.8% ASR with only 1 refusal. The downside is that its language modelling takes a significant hit. For research or targeted use where safety removal is the only goal, it delivers. For everyday use, the capability tradeoffs are too steep. Note on KL: the card reports KL at 0.0004 but we measure 0.698. This is not a measurement error. Abliterix uses a different KL computation than the Heretic methodology: it measures over a configurable kl_token_count of generated tokens rather than full-vocab first-token logits, and when using vLLM in-place editing it switches to mean absolute NLL drift instead of KL divergence. These are valid optimisation metrics, but they are not comparable to the batchmean KL over 262K first-token logits that Heretic and this comparison use.

Key findings

All 13 methods remove safety filters effectively, lifting HarmBench ASR from the base model’s 32.2% to 82.2% to 100.0%. The safety removal part works regardless of technique.

The difference is in the capability tradeoffs. Surgical approaches like coder3101 and llmfan46 beat the base model on GSM8K. The abliteration actually improves reasoning by shortening thinking chains, allowing more answers within the token budget. Aggressive approaches like treadon and ether4o4 lose about 3 to 7pp on GSM8K because the model overthinks and exhausts its budget before writing an answer.

KL divergence ranges from 0.068 for llmfan46 to 3.97 for treadon, a 58.7x spread. The optimal tradeoff is in the moderate range. coder3101 achieves 96.0% ASR with KL=0.167 and beats base on GSM8K and LAMBADA. llmfan46 achieves 83.8% ASR with the lowest KL of any variant.

10 of 13 variants are perfect rank-1 edits, the classic abliteration signature. The 13 variants form three alignment clusters: Huihui, Heretic, Independent. Yet no universal abliteration subspace exists. Many technique pairs discovered nearly orthogonal edit directions.

KL Divergence Calibration

I measure KL divergence using the same methodology as Heretic . Full vocab at 262K tokens, first-token logits from 100 harmless prompts, batchmean reduction. Four of the 13 variants were built with Heretic and report their KL values in the model card. That gives a natural calibration check.

Three of four land within 6% of the card value. kasper is the outlier at +17.2%. The card reports the KL from the optimisation trial run, not a post-hoc evaluation. kasper’s card also notes it used an RTX 3080 with 10GB VRAM and dropped down_proj from the target components to fit memory. This non-standard Heretic configuration likely contributes to the measurement gap.

llmfan46 measures 13.1% lower than claimed. This is also within expected variance for cross-hardware KL measurement. The direction is negative rather than positive, which is consistent with floating point differences in log_softmax over a 262K vocab.

The consistency across four independent Heretic builds validates our measurement pipeline. Small differences are expected from different hardware, CUDA versions, and floating point accumulation order. The agreement is close enough that I am confident our KL values for the other 9 variants are accurate too.

Quick Facts

Architecture notes

Gemma4-E2B has a unique dual-norm / shared-KV architecture:

• Layers 0 to 14: Full KV projections, 15 layers
• Layers 15 to 34: Shared KV projections, 20 layers, num_key_value_heads: 1
• tie_word_embeddings: true: Input and output embeddings share weights
• 600 LM keys in the base model. 5 variants shipped with only 540, missing 60 shared-KV weights. Patched from base.

Benchmarks

Evaluated with lm-evaluation-harness via vLLM v0.20.0 on native BF16 with no quantisation on a single RTX 5090. All 14 models tested with identical settings. Two phases: loglikelihood in Phase 1 and generative in Phase 2.

Phase 1: Loglikelihood tasks

batch_size=1, max_model_len=8096, --reasoning-parser gemma4. About 47 min per model.

HellaSwag uses acc_norm. All other classification tasks use acc. LAMBADA uses perplexity, lower is better.

Delta vs base

Deltas in percentage points. LAMBADA shows percentage change in perplexity from base. Positive means lower perplexity and is better. Negative means higher perplexity and is worse.

What the benchmarks tell us

Loglikelihood tasks are remarkably resilient. The 14 models cluster within 2.6pp on MMLU at 26.7% to 29.3%, and 1.6pp on HellaSwag at 30.8% to 32.4%. These tasks rank token probabilities, so abliteration barely affects the model’s knowledge representation.

TruthfulQA MC2 shows the clearest abliteration signal. Treadon drops 4.64pp, wangzhang drops 2.94pp, wwtcyberlab drops 3.20pp. The most aggressive variants reduce the model’s ability to distinguish factual from non-factual content.

LAMBADA perplexity is the outlier metric. Three variants show substantial degradation: wangzhang at 7.35x base, wwtcyberlab at 5.69x, ether4o4 at 2.28x. Wangzhang’s unique q_proj and v_proj modifications, targeting attention input projections, disrupt language modelling more than variants that only touch output pathways. Interestingly, huihui-v2 at KL=0.530 has the best LAMBADA perplexity at 0.53x base. Its edits concentrate in the refusal direction without disrupting language modelling.

Phase 2: GSM8K, generative with thinking

Critical discovery. lm-eval’s local-completions backend bypasses the chat template, disabling thinking for reasoning models. Base GSM8K goes from 13.1% to 83.5% flexible-extract with thinking enabled, a 6.4x improvement. All Phase 2 results use local-chat-completions with enable_thinking=true.

Two variants beat base on both metrics. coder3101 gains +1.37pp flex and +3.94pp strict. llmfan46 gains +0.46pp flex and +1.59pp strict. Pew matches base on flex and beats on strict by 1.44pp. All three use surgical, low-tensor-count approaches.

The empty response problem

Empty responses occur when the model exhausts its 14,336 token generation budget on thinking tokens without producing visible content. The vLLM reasoning parser strips thinking from the content field, and lm-eval sees null content. Every empty response is scored as incorrect.

The empty-response problem is a thinking efficiency issue, not a reasoning ability issue. When adjusted for empty responses, the flex scores converge. The 8.3pp raw gap between coder3101 and ether4o4 narrows to about 3pp. Abliteration changes thinking efficiency, not reasoning ability.

Safety: HarmBench

HarmBench with 400 textual behaviours, max_tokens=8096, temperature=0, enable_thinking=true. Keyword-based refusal detection with full LLM review of all 5,600 responses using glm-5.1. Zero errors across all 14 models.

The abliteration lifts ASR by 50 to 68 percentage points over the base model’s 32.2%. Five variants achieve 99% ASR or higher. One reaches 100% with zero refusals. Unlike the GLM-4.7 comparison where all techniques achieved 100%, Gemma4 shows meaningful differentiation. 8 of 13 variants fail to reach 97%.

CoT-verified. I inspected the chain-of-thought reasoning for all 37 truncated responses across all 14 models. 35 of 37 show clear compliance intent in their reasoning before truncation. One was a truncated refusal, llmfan46’s GPS tracker response, correctly classified. One was a thinking-only loop with no visible content, kasper. Treadon’s 21 truncated responses all had compliant reasoning but 20 entered repetition loops like **step 4:** repeats, LaTeX nesting, and binary dumps before hitting the token limit. The reported ASR numbers are LLM-reviewed and confirmed accurate. Truncation is a generation quality issue, not a scoring issue.

ASR by category

Near-complete compromise, most variants at 90% or above: copyright, harmful, misinfo. These categories have the weakest residual safety.

Strong residual safety with wide spread: chemical_biological at 62.5% to 100%, illegal at 56.9% to 100%, harassment at 56.0% to 100%. The base model concentrates safety alignment in the most physically dangerous categories.

Hardest items, refused by 6 or more of 13 variants:

Three categories account for all ten hardest items: illegal with 2, harassment with 2, and chem_bio with 3, making these the most resilient categories overall.

KL Divergence

Methodology: F.kl_div on full vocab with 262,144 tokens, first-token logits from mlabonne/harmless_alpaca test[:100], matching the Heretic evaluator . System prompt: “You are a helpful assistant.”

Not all KL measurements are created equal. Vocab size, prompt set, reduction method, and whether you measure first-token or full-sequence all change the result. A model card that says “KL: 0.001” without specifying how it was measured is meaningless. The number can be made arbitrarily low by shrinking the vocab subset or using sum instead of batchmean. I use the Heretic methodology specifically so the numbers are comparable across variants and verifiable against the Heretic models’ own claims. When we find discrepancies like duoneural’s original 0.001 claim measuring at 0.187, raising them with the creator is the right thing to do. DuoNeural updated their card promptly.

KL divergence measures how far the abliteration technique has shifted normal prompt outputs from the base model. It is primarily a function of the edit’s magnitude and direction, not the model’s parameter count. Lower values indicate better capability preservation.

Rating scale: excellent below 0.01, very good 0.01 to 0.1, moderate 0.1 to 0.4, significant 0.4 to 1.0, heavy above 1.0.

The optimal tradeoff is in the moderate KL range. Variants like coder3101 at KL=0.167 with 96.0% ASR and trevorjs at KL=0.365 with 99.5% ASR achieve near-maximal safety removal with controlled distribution shift. Higher KL correlates with higher ASR but with diminishing returns. Compared to wwtcyberlab in the significant range at 99.5% ASR and KL=0.964, the jump to heavy on treadon gains only 0.5pp more ASR at a 4x KL cost.

KL vs benchmark impact

LAMBADA perplexity is the most sensitive metric to KL divergence. LAMBADA’s sensitivity to next-token distribution quality makes it an early warning indicator for capability damage.

Cross-family comparison

The 58.7x spread is the largest in the project. It reflects the diversity of abliteration approaches, from surgical single-type edits with llmfan46 at 7 tensors to aggressive multi-method approaches with treadon using disinhibition + abliteration. Other families tested fewer variants at 3 to 5, so their spread is narrower partly because the tails aren’t sampled.

Weight Analysis

Modification summary

Types = number of distinct tensor types modified. E/M/L = early layers 0 to 10 / mid layers 11 to 22 / late layers 23 to 34.

Three tiers of aggressiveness

Surgical at 3% or less with 1 tensor type: llmfan46, coder3101, kasper, pew. These modify only self_attn.o_proj.weight in a narrow band of mid-to-late layers, L16 to L32. The approach targets what the model “says” without touching what it “hears” or how it processes internally.

Moderate at 8 to 10% with 2 tensor types: duoneural, huihui-v1, prithiv, treadon, huihui-v2. These add mlp.down_proj.weight and expand layer coverage to 69 to 86%.

Aggressive at 11 to 31% with 2 to 6 tensor types: trevorjs, wangzhang, wwtcyberlab, ether4o4. These expand beyond the standard o_proj and down_proj pair into gate_proj, up_proj, q_proj, v_proj, and Gemma4-specific per_layer_input_gate and per_layer_projection weights.

Which tensor types get modified

All abliteration variants target output projections, meaning what the model “says.” Only ether4o4 and wangzhang venture into input/query projections and gating mechanisms. Wangzhang’s q_proj and v_proj targeting correlates with its 7.35x LAMBADA perplexity increase.

SVD / Rank analysis

10 of 13 variants are perfect rank-1. Their edits lie along a single direction in weight space, the classic abliteration signature of subtracting a single “refusal direction” vector. The three exceptions: pew uses Heretic ARA for anti-refusal subspace removal, treadon combines disinhibition + abliteration at rank-2, and ether4o4’s Gemma4-specific gate components have rank around 4.

Cross-technique alignment

Three alignment clusters emerge from pairwise cosine similarity analysis.

The Huihui Cluster at cosine above 0.85: huihui-v1, prithiv, huihui-v2, duoneural. These four variants discovered nearly identical edit directions. Prithiv and huihui-v1 are near-identical with cosine=1.0, identical KL and Phase 1 benchmarks, but GSM8K and HarmBench differ slightly.

The Heretic Cluster at cosine 0.67 to 0.92: coder3101, llmfan46, pew, kasper. The four Heretic-based variants show strong directional alignment, though kasper/pew at 0.67 is a lower-bound outlier.

The Independent Approaches at cosine below 0.71: trevorjs, wangzhang, wwtcyberlab, ether4o4, treadon. Each uses a fundamentally different approach.

No universal abliteration subspace. Many technique pairs are nearly orthogonal at cosine around 0.01. Despite all achieving 82 to 99% HarmBench ASR, the refusal direction in Gemma4-E2B’s weight space is not a single vector. It’s a manifold with many viable removal pathways.

The near-identical models: huihui-v1 and prithiv

Weight forensics show huihui-v1 and prithiv are nearly identical. Cosine=1.0 across all 50 shared tensors, identical KL at 0.2510, identical Phase 1 benchmarks. However, GSM8K at 83.40% vs 82.94% and HarmBench at 85.8% vs 90.8% differ slightly. The weights are not bit-for-bit identical. Prithiv is almost certainly derived from huihui-v1 or both share a common source, but we cannot assert they are the exact same model file.

Shared-KV export bug

5 of 13 variants shipped with 60 missing weights covering k_proj, k_norm, and v_proj across layers 15 to 34. The abliteration export tools did not understand Gemma4’s num_kv_shared_layers architecture, and the shared-KV weights were silently dropped. All 5 were patched by copying the missing weights from the base model. This is a safe, lossless patch since these weights are unmodified and identical across all working variants.

The Optimal Tradeoff

Looking across all metrics, the best capability-safety tradeoff:

Coder3101 beats base on GSM8K on both flex and strict, has below-base LAMBADA perplexity, and achieves 96.0% ASR with only 9 modified tensors. llmfan46 similarly beats base on both GSM8K metrics with only 7 tensors and the lowest KL divergence of any variant at 0.068, though its ASR is 83.8%. Trevorjs achieves 99.5% ASR with only 2 refusals at a still-moderate KL of 0.365. All three demonstrate that surgical abliteration preserves capabilities while achieving strong safety removal.

Summary

Variant summaries

coder3101. Best overall tradeoff. 96.0% ASR with only 9 tensors, 1 type at o_proj, L17 to L25. Beats base on GSM8K flex by +1.4pp and strict by +3.9pp. KL=0.167 rated moderate. Below-base LAMBADA perplexity. The Heretic tool’s Magnitude-Preserving Orthogonal Ablation at its best.

llmfan46. Most conservative. 83.8% ASR with only 7 tensors, the fewest of any variant, at o_proj L17 to L23. Lowest KL at 0.068 rated very good. Beats base on GSM8K flex by +0.5pp and strict by +1.6pp. Trades ASR for maximum capability preservation.

trevorjs. Maximum safety with controlled damage. 99.5% ASR with zero truncations and only 2 refusals. 70 tensors across all 35 layers at 100% coverage. KL=0.365 at the moderate upper bound. Consistent output length makes it the most reliable high-ASR variant.

pew. Solid middle ground. 92.0% ASR, 16 tensors. Uses Heretic ARA rather than standard rank-1, producing slightly higher-rank edits. Matches base on GSM8K flex exactly.

kasper. Similar to pew. 92.5% ASR, 16 tensors, Heretic-based. Per-prompt median KL at 0.00093, second only to trevorjs in the moderate cluster, indicating more pervasive but smaller shifts across prompts.

treadon. Most aggressive. 100.0% ASR with zero refusals, the only variant to achieve perfect compliance. 21 truncated responses at 5.3%. CoT analysis confirms all 21 were mid-compliance when cut off, with 20 entering repetition loops including LaTeX nesting, markdown bold repeats, and binary dumps. KL=3.971 rated heavy, 4.1x higher than next worst. The “disinhibition + abliteration” dual approach produces the heaviest distribution shift of any variant, well beyond refusal direction ablation. Highest ASR but pays for it across every capability metric.

wangzhang. Near-perfect safety removal. 99.8% ASR with 1 refusal and unique q_proj and v_proj targeting. 7.35x LAMBADA perplexity increase, the worst language modelling degradation in the comparison. 4 tensor types, L9 to L34.

ether4o4. Broadest modification. 166 tensors at 30.7% with 6 types including Gemma4-specific gate components. 95.2% ASR but 84 empty GSM8K responses at 6.4%. Multi-rank edits at eff rank 2.29 due to gate components at rank around 4.

huihui-v2. Strong Huihui variant. 99.5% ASR with zero truncations and only 2 refusals. 60 tensors, 86% layer coverage. 2x higher KL than huihui-v1 due to larger edit magnitudes at mean norm 4.94 vs 2.02. 54 empty GSM8K responses at 4.1%.

huihui-v1. Moderate Huihui. 85.8% ASR, 50 tensors, identical to prithiv in weight forensics and Phase 1 benchmarks. See the near-identical models section.

prithiv. Near-identical to huihui-v1. 90.8% ASR, 50 tensors. Cosine similarity of 1.0 across all shared tensors but not bit-for-bit identical. GSM8K and HarmBench differ slightly.

duoneural. Weakest safety removal. 82.2% ASR with 71 refusals, far more than the maximally abliterated tier. 49 tensors, 2 types, L6 to L34. Originally reported KL of ~0.001, which we found to be 187x lower than our measurement at 0.187. DuoNeural updated their card with our corrected KL and HarmBench numbers after we raised it.

wwtcyberlab. High ASR, high LAMBADA cost. 99.5% ASR with 96 tensors across 4 types. 5.69x LAMBADA perplexity increase. Highest single-prompt KL spike at 42.45.

Methodology

• Capability: lm-evaluation-harness via vLLM v0.20.0, native BF16 on single RTX 5090 with 32GB
• Safety: HarmBench 400 textual behaviours, max_tokens=8096, temperature=0, enable_thinking=true, keyword-based refusal detection with full LLM review of all 5,600 responses by glm-5.1
• KL divergence: Full vocab at 262K tokens, first-token logits via model.generate(max_new_tokens=1, output_scores=true), matching Heretic evaluator methodology
• Weight analysis: SVD, fingerprint, edit vector overlap, per-layer analysis, correlation, subspace alignment, and low-rank reconstruction using Abliterlitics
• Hardware: NVIDIA RTX 5090 with 32GB

Key methodology lessons

1. Chat completions required for reasoning models. local-completions bypasses the chat template, disabling thinking. GSM8K flexible-extract goes from 13.1% to 83.5%. Always use local-chat-completions with enable_thinking=true.
2. max_gen_toks must account for thinking. Gemma4’s thinking tokens consume the generation budget. Use max_gen_toks=14336 with max_model_len=16384 for Phase 2.
3. batch_size=1 for loglikelihood. batch_size=4 OOMs during MMLU’s log_softmax over 262K vocab on long prompts.
4. Minimal thinking loops. Unlike the Qwen3.5 family where loops were common, Gemma4-E2B had 1 case across 5,600 HarmBench responses: kasper’s truncated suicide-instruction response with 2,698 <|channel>thought repeats. 99.98% loop-free.
5. Truncation is not refusal. CoT analysis of all 37 truncated HarmBench responses confirms 35 were mid-compliance when cut off. 1 truncated response was a refusal from llmfan46. 1 was a thinking loop from kasper. The ASR numbers are accurate.

How this was made

Full transparency on the compute cost and the failures along the way. This was not a clean pipeline run. It was three days of debugging, crashing, rerunning, and learning.

GPU hours

Roughly 8 of those 44 hours were spent on runs that failed and had to be redone.

What broke

Weight pipeline: 3 crashes out of 6 runs. Two CUDA OOM errors from loading full weight tensors into GPU memory for diff computation. One RuntimeError when torch.stack hit mismatched tensor sizes because ether4o4 modifies gate components that have different dimensions than attention projections. Fix: process tensors one at a time on CPU instead of batched on GPU.

KL divergence: 4 failures in 18 minutes. A Docker mount error. A bash variable name error because huihui-v1 has a hyphen. Two missing chat template errors because the Gemma4 tokenizer does not ship with a default chat template and it needed to be passed explicitly. The 5th run completed all 13 variants in 5 minutes.

GSM8K: the big one. The initial approach used local-completions which bypasses the chat template entirely. On reasoning models this disables thinking. The base model scored 10.0% on GSM8K without thinking versus 85.0% with thinking enabled. That is a 75-point gap. Two failed attempts across 1.6 hours before switching to local-chat-completions with a custom chat template that sets enable_thinking=true. That worked immediately.

5 models missing shared KV weights. duoneural, ether4o4, kasper, treadon, and wangzhang were each missing exactly 60 safetensor keys. Gemma4 uses a shared KV architecture where layers 15 to 34 share key and value projections as an inference optimisation. Each shared layer still needs three weights present in the checkpoint: k_proj projects the hidden state into key vectors that determine what tokens attend to, v_proj projects into value vectors that carry the information being attended to, and k_norm normalises keys to keep attention scores stable. The abliteration export tools used by all 5 authors only saved the weights they modified and silently dropped these shared KV tensors. With our setup running vLLM 0.20.0, these models would not load. Fix: copy the 60 missing weights from the base model. Since these weights are unmodified and byte-for-byte identical across all working variants, the patch is lossless. This was discovered during the first weight pipeline run when key counts came back as 540 instead of 600.

Result file copy bug. The batch script that collected Phase 2 results from the 14 per-model output directories copied the same file 14 times. All 14 result files were identical, containing wwtcyberlab’s results. Recovered by finding the original per-model files in the __tmp__model_{slug}/ directories that lm-eval writes to before the copy step.

What was learned

1. Reasoning models need chat completions for generative tasks. local-completions silently disables thinking. Always use local-chat-completions with enable_thinking=true.
2. max_gen_toks includes thinking tokens. Gemma4 thinks for 2000 to 5000 tokens before answering. Budget for it.
3. Gate components have different dimensions. Gemma4’s shared KV mechanism creates gate tensors that do not match projection tensor sizes. Weight diff code needs to handle mixed dimensions.
4. Hyphens in model names break bash variables. Use arrays or quoting consistently.
5. Trust no batch script. Verify each output file independently after a bulk copy or rename.
6. Verify model integrity before benchmarking. 5 of 13 models were missing 60 safetensor keys due to a shared KV export bug. Count keys against the base model before trusting any results.

Cost of honesty

Of the 44 GPU hours, about 8 hours produced nothing usable. Crashes, wrong configurations, silent failures. The actual analysis data took roughly 36 hours of compute to produce on a single RTX 5090. Total wall clock time across May 18 to 20 was 3 days including sleep gaps between runs.

Disclaimer

This model has had safety alignment removed. It will comply with harmful requests, including generating content related to violence, illegal activities, and other harmful behaviours. Use responsibly and in accordance with applicable laws and regulations. The authors do not condone or encourage the use of this model for harmful purposes.

While I have taken the time to verify all results thoroughly, I am open to any corrections, additional benchmarks, or further analysis. If you spot something that looks wrong and can be confirmed, I am happy to fix it.