What is Abliterix?

Abliterix is a Heretic-derived multi-objective Optuna optimiser with native hybrid Mamba/attention support, projected abliteration, and expert-granular steering. It applies rank-3 LoRA-merged directional updates to attn.o_proj and mlp.down_proj.

Performance Across Models

I’ve tested Abliterix on Qwen3.6-27B, GLM-4.7-Flash, and Gemma4-E2B.

Gemma4-E4B uses MMLU-Pro from the mixed v1/v2 suite. LAMBADA is excluded from the e4b suite as a raw-completion task, so no perplexity figure is reported.

Key Characteristics

Quantisation sensitivity. Under BNB4 quantisation, Abliterix shows the worst capability preservation of any technique. Lambada perplexity increases 2.9x from 3.18 to 9.12 on Qwen3.6-27B. On Gemma4-E2B, the LAMBADA perplexity blowup is even worse at 7.35x base, reaching 1,072,918. The q_proj and v_proj modifications unique to the Gemma4 variant catastrophically damage language modelling. The model’s creator explains this as a quantisation interaction. The rank-3 LoRA signal lives in a low-dimensional subspace, and BNB4’s per-block NF4 quantisation is not subspace-aware.

Surgical component targeting. Despite the broad-sounding “LoRA search” description, Abliterix only modifies attn.o_proj and mlp.down_proj on Qwen3.6-27B and GLM-4.7-Flash. Just 2 weight types. It does this across all layers with a mid-to-late-stack sustained edit profile.

Gemma4-E2B attention targeting. The Gemma4 variant uniquely modifies q_proj and v_proj in addition to the standard attn.o_proj and mlp.down_proj. This is the only variant in the entire Gemma4 comparison that targets query and value projections. The attention input targeting correlates with catastrophic LAMBADA degradation at 7.35x base.

Mid-to-late-stack edit profile. The abliteration weight peaks at layer ~41 with a 35-layer decay radius, creating a sustained modification floor across the middle and late layers.

Gemma4-E4B hit perfect ASR with mixed-attention targeting. This is the variant that answered all 400 HarmBench behaviours with zero refusals, zero empty responses, and zero truncations. KL landed at 0.0536, the very good band. On Gemma4, Abliterix modified 89 tensors across 4 types: o_proj, q_proj, k_proj, and down_proj. It was the only variant in the 23-way comparison to target query and key projections. Unlike the e2b wangzhang variant that catastrophically damaged language modelling by touching q_proj and v_proj, the e4b run kept MMLU-Pro within 0.5pp of base and GSM8K within 0.2pp. The cost showed up on IFEval at -4.4pp, the clearest instruction-following signal. So mixed-attention targeting can reach 100% ASR without the language-modelling collapse seen on e2b, but it still taxes instruction following.

Card KL under-reports by 89x. The e4b model card claims KL of 0.0006. I measured 0.0536. This is not a measurement error. Abliterix uses a different KL computation than the Heretic methodology. It measures over a configurable kl_token_count of generated tokens rather than full-vocab first-token logits, and under vLLM in-place editing it switches to mean absolute NLL drift. These are valid optimisation metrics, but they are not comparable to the batchmean KL over 262K first-token logits that this project and Heretic use.

Weight Modification Profile

• 101 tensors modified (11.9% of total)
• 5.2% relative edit magnitude
• Components: attn.o_proj + mlp.down_proj across all 64 layers
• Profile: peak at layer ~41, 35-layer decay radius

Read the Full Analyses

• Qwen3.6-27B: Heretic vs Huihui vs AEON vs Abliterix vs HauhauCS
• GLM-4.7-Flash: Heretic vs Huihui vs HauhauCS vs Abliterix
• Gemma4-E2B: 13 Abliteration Techniques Compared
• Gemma4-E4B: 23 Abliteration Variants Compared

External Links

• Abliterix on HuggingFace
• Abliterix v1.4 discussion