Table of Contents
What is Apostate?
Apostate
is an abliteration tool by heterodoxin that uses weight-space orthogonal projection with configurable profiles. The balanced profile targets both o_proj and down_proj across nearly all layers with moderate edit intensity. It is a new tool, first benchmarked in this project on Qwen2.5-7B. On Gemma4-E4B it surprised me. The same tool took a completely different path through the model, editing the MLP feed-forward block and skipping attention entirely.
Performance Across Models
| Model | HarmBench ASR | MMLU | KL Divergence | Avg Delta (excl GSM8K) |
|---|---|---|---|---|
| Qwen2.5-7B | 98.8% | 71.43% | 0.134 | -1.0pp |
| Gemma4-E4B | 85.8% | 43.97% (MMLU-Pro) | 0.0036 | -0.4pp |
The Gemma4-E4B row uses MMLU-Pro from the mixed v1/v2 suite, not MMLU.
Key Characteristics
Broad layer coverage with moderate intensity. Apostate modified 55 tensors across 27 of 28 layers on Qwen2.5-7B, skipping only layer 11. Mean edit norm of 1.63 is gentler per tensor than Heretic at 2.33.
Near-zero direction overlap with Huihui. Despite targeting the same tensor types on the same base model, Apostate and Huihui have cosine similarity of just 0.023 on Qwen2.5-7B. They found almost entirely different refusal directions yet achieved nearly identical ASR. This confirms the safety subspace has multiple independent removal paths.
Lowest KL batchmean on Qwen2.5-7B. At 0.134, Apostate has the lowest KL divergence of the three variants tested. The most balanced distribution shift.
Three-phase layer pattern. Low edits on layers 0 to 5, high on 6 to 20, reduced on 21 to 27. This suggests Apostate concentrates edits where the refusal direction is most strongly represented.
Embedding touch. Like Huihui, Apostate touches the embedding with minimal norm on Qwen2.5-7B. Heretic skips it entirely.
Gemma4-E4B goes through the MLP, not attention. This is the standout finding. On Gemma4-E4B, Apostate modified 152 tensors across mlp.down_proj, mlp.gate_proj, mlp.up_proj, and per_layer_input_gate. Zero o_proj. Zero q_proj, k_proj, v_proj. It never touched attention. Yet KL landed at 0.0036, GSM8K strict came in within 0.5pp of base, and MMLU-Pro within 0.05pp. It posted the lowest capability damage of any variant in the whole 23-way comparison. The tradeoff is ASR at 85.8%, below the Heretic cluster, with the gap concentrated in chemical/biological and illegal categories. Apostate proves refusal is reachable through the MLP feed-forward block, not just attention. MLP edits spread across parallel layers, which makes collateral damage less likely than attention edits concentrated in a single sequential path.
Weight Modification Profile
Qwen2.5-7B:
- Targets 2 weight types:
self_attn.o_proj.weightandmlp.down_proj.weight - 55 of 339 tensors changed (16.2%)
- 2.72B parameters changed (35.8%)
- Layers modified: 27 of 28
- Peak edit at layer 18
Gemma4-E4B:
- Targets 4 weight types, all MLP:
mlp.down_proj,mlp.gate_proj,mlp.up_proj, andper_layer_input_gate - 152 of 719 tensors changed (21.1%)
- Layers modified: all 42
- No attention weights touched