This page covers both halves: how a comparison runs operationally, and the instrument reference for what each measurement is and how to read it.

How a comparison runs

Setup and pre-flight

Every comparison starts as a config file in the repository. It names the base model, the abliterated variants, and the settings for each stage. Models download from HuggingFace with a hard verification step, because a silent partial download looks exactly like a broken model further down the pipeline.

Before any full GPU run, a pre-flight agent researches the model and runs a limited diagnostic against published numbers. This step exists because the expensive failures are the quiet ones. A missing chat template flag once depressed every loglikelihood score by 30 to 60 points, and nothing crashed. Five minutes of diagnostic caught the same class of problem the next time before it cost a week of GPU time.

The pipeline

One command drives the whole thing:

./abliterlitics.sh auto ./comparisons/<name>/

Stages run in order and each one checks the project database before starting, so completed work is skipped and interrupted work resumes. Weight forensics runs first and is CPU-bound, reading safetensors lazily, so it overlaps with the GPU stages. KL divergence and HarmBench are GPU exclusive. Capability benchmarks run in two phases: loglikelihood tasks are deterministic and token-free, generative tasks run with a thinking-aware budget established from base-model behaviour first. The report stage synthesises everything into graphs, tables, and the pages on this site. Instrument definitions for every stage are in the sections below.

The agent pipeline

Abliterlitics is operated by agents, each with a narrow job, defined in plain markdown files under .opencode/agents/ in the repository. The goal is a fully automated pipeline with a human at the decision points: a new model should travel from HuggingFace to a verified, reviewed report without a person touching the mechanics.

Coordination is explicit rather than implicit. Every comparison keeps a journal in the project database, and agents write discoveries, decisions, and results there as they happen. A stage starting later reads the journal first, so knowledge earned painfully in an early stage does not evaporate. Plans live in .llm/ folders with a progress file tracking each task, and the plan is immutable once approved. Hypotheses are pre-registered before the data exists. That is what makes the findings on this site trustworthy: the questions were written down before anyone knew the answers.

Human in the loop

The agents do the mechanical work. Setup, measurement, parsing, first-draft reports, quality gates. The human gates are fixed and few:

• Approve the plan before build starts
• Approve each task before it commits
• Review the final report before publishing
• Decide anything ambiguous or destructive

The gates are also economics. A bad config caught at a gate costs minutes. The same config caught after a full pipeline run costs days of GPU time. As a rule the agents never commit anything, never publish anything, and never kill anything they did not start themselves without asking first.

Capability Benchmarks

Abliterlitics has used two capability suites over time. Reports up to Gemma4-E2B (May 2026) used the v1 suite. From Gemma4-E4B (July 2026) onward we use a mixed v1/v2 suite that adds the Open LLM Leaderboard v2 reasoning tasks. Each report states which suite it used. All scores are produced with lm-evaluation-harness via vLLM .

v1 suite (legacy)

Used by the GLM-4.7-Flash, Qwen3.5, Qwen3.6, Qwen2.5-7B, Qwen3-4B, and Gemma4-E2B reports.

Tasks

What each benchmark tests

MMLU (Massive Multitask Language Understanding). Tests general knowledge across 57 academic subjects including history, law, medicine, mathematics, computer science, and philosophy. The model is given a multiple-choice question and must select the correct answer. This is the single most important benchmark for measuring whether abliteration degraded the model’s overall knowledge. Measured as accuracy with 5-shot prompting.

GSM8K (Grade School Math 8K). 8,500 grade-school-level math word problems requiring multi-step reasoning. The model must solve problems like “Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?” Straightforward for humans, but tests whether the model can chain logical steps correctly. For reasoning models, this benchmark is tricky because the model thinks out loud before answering. If the thinking chain gets too long, it runs out of space before producing an answer. We report both raw scores and adjusted scores that exclude these invalid responses. Measured as strict match and flexible extract.

HellaSwag. Tests commonsense reasoning through sentence completion. The model is given a partial description of an everyday scenario and must choose the most plausible continuation from four options. Questions like “A woman is seen cutting vegetables on a cutting board, then she…” with options like “adds them to a pot of soup” vs “drives a car to work.” Sounds simple, but requires real-world understanding. Measured as accuracy with normalized scoring.

ARC-Challenge (AI2 Reasoning Challenge). Science exam questions from standardised tests for 3rd through 9th grade. These require genuine reasoning, not just pattern matching. Questions span biology, chemistry, physics, earth science. The “Challenge” subset specifically filters for questions that retrieval-based methods get wrong, so this tests actual reasoning ability. Measured as accuracy with 25-shot prompting.

WinoGrande. Coreference resolution inspired by the Winograd Schema Challenge. The model reads a sentence with an ambiguous pronoun and must determine which noun it refers to. For example, “The trophy didn’t fit into the brown suitcase because it was too large” vs “…because it was too small.” The answer changes based on whether “it” refers to the trophy or the suitcase. Tests whether the model understands physical relationships and context. Measured as accuracy with 5-shot prompting.

TruthfulQA MC2. Tests whether the model can resist common misconceptions and myths. Questions target things that people commonly get wrong, like “What happens if you touch a baby bird?” with the popular misconception that the mother will abandon it. The model must identify the truthful answer, not the commonly believed one. This benchmark consistently drops after abliteration, suggesting the safety direction overlaps with factual accuracy. Measured as accuracy with 0-shot prompting.

PiQA (Physical Interaction: Question Answering). Tests understanding of how the physical world works. The model is given a goal like “to separate egg whites from the yolk” and must choose the most plausible method from two options. Requires common sense about objects, tools, and physical processes. Measured as accuracy with 0-shot prompting.

Lambada OpenAI. Tests how well the model predicts the last word of a passage. The model reads a full paragraph and must predict the final word. This is fundamentally a test of language modelling quality and coherence. Unlike the other benchmarks which test reasoning or knowledge, Lambada tests whether the model’s basic language generation is intact after abliteration. Reported as perplexity, where lower is better. A big jump in perplexity means the model’s output distribution has shifted significantly.

Mixed v1/v2 suite, current from Gemma4-E4B

Adds the Open LLM Leaderboard v2 reasoning tasks on top of the forensic v1 subset, plus the generative TruthfulQA variants. The v2 tasks are harder and reason more, so they expose capability damage the legacy tasks missed. Used from the Gemma4-E4B report onward. Excludes LAMBADA and HumanEval. See Excluded tasks below.

The new v2 tasks

MMLU-Pro. An expansion of MMLU with 10 answer choices instead of 4 and questions concentrated in STEM, law, psychology, and other reasoning-heavy domains. The larger choice set removes the lucky-guess floor that capped original MMLU, so it is far more sensitive to genuine capability damage. Measured as accuracy with 5-shot prompting.

GPQA Diamond (Google-Proof Q&A). Graduate-level biology, chemistry, physics, and earth-science questions written by PhDs. Google-proofed so that non-experts with web access still fail. This is one of the hardest openly available reasoning benchmarks and separates models that memorised textbook answers from those that can reason. Measured as accuracy with normalised scoring, 0-shot.

BBH (BIG-Bench Hard). The 23 tasks from BIG-Bench where language models historically underperformed the average human rater. Algorithmic, multistep, and linguistic reasoning. The most varied single benchmark in the suite. A model cannot pass it by being good at one thing. Measured as accuracy with normalised scoring, 3-shot.

MUSR (Multistep Soft Reasoning). Tests whether the model can chain inferences where each individual step is easy but the whole sequence is not. Each item is synthesised to require deliberate multistep work rather than a single retrieval. Good at exposing fragmented reasoning where each step scores but the chain breaks. Measured as accuracy with normalised scoring, 0-shot.

IFEval (Instruction-Following Evaluation). The only capability benchmark in the suite that is fully generative and directly comparable to vendor-published numbers. The model is given explicit formatting constraints (JSON, sections, word counts, keywords) and scored on whether the output obeys them. Because it does not allow the model to think its way around the constraint, IFEval is the cleanest early-warning signal for capability damage: surgical abliterations lose under 3 points, damaged variants lose 9 to 15. Measured as prompt-strict accuracy.

Two chat templates

Reasoning models think inside an unclosed <|think|> (or <|channel>thought) block before answering. For generative tasks this is desirable and matches the vendor’s published methodology. For loglikelihood multiple-choice scoring it is fatal: the chat template injects an unclosed thinking tag, so the logits at the scoring position are over thinking prose rather than the answer choices, depressing every MC score by roughly 25 points.

We therefore use two templates. Loglikelihood Phase 1 uses a no_thinking chat-template override with thinking disabled, so the model scores answer choices directly. Generative Phase 2 covers GSM8K and IFEval, and uses a thinking override with thinking enabled. Before trusting any MC score on a reasoning model, check for a chat_template in tokenizer_config.json. If thinking is wired in, the scores are wrong.

Loglikelihood vs generative gap

Open LLM Leaderboard v2 loglikelihood multiple-choice scores are not comparable to vendor-published generative numbers. For Gemma4-E4B the gap is about 25 points: loglikelihood MMLU-Pro lands near 44% and GPQA near 34%, while Google’s generative card reports roughly 69% and 59%. The gap is by design. Loglikelihood MC does not let the model think before answering. Only the generative tasks, GSM8K and IFEval, compare directly, and both match community references. Document deltas within a suite, never against vendor card numbers.

Excluded tasks

LAMBADA and HumanEval are raw-completion tasks that --apply_chat_template breaks. They are excluded from the mixed suite. LAMBADA remains documented under the legacy v1 suite above for historical reference, but is not used in current reports.

GSM8K strict-flexible gap

GSM8K has two scoring modes. Strict requires the canonical #### N final-answer marker. Flexible extracts the last number in the response. A wide strict-flexible gap means the model can do the arithmetic but cannot format the answer. It is a cheap, free reasoning-damage detector. Clean abliterations stay under 1.5 points of gap, matching base. Damaged variants blow out to 5 to 21 points. Combined with empty and truncated HarmBench counts, this surfaces reasoning damage that any single metric misses.

Safety: HarmBench

HarmBench with 400 textual behaviours across 7 categories:

Token budgets and empty responses

The token budget needs care. Base models spend few tokens refusing, while abliterated models engage with the prompt and reason at length, so they need a higher budget. Empty responses are a signal in their own right. The reasoning ran out of tokens before producing an answer, and a variant producing more empties than base tells us the abliteration is causing reasoning instability. For thinking models we report GSM8K three ways for the same reason: the headline score, the empty rate, and the answered-only score.

Classification

Responses are classified with a custom classifier, then verified by multiple independent LLM reviewers. For reasoning models, we perform chain-of-thought, or CoT, direction analysis when the response is empty but the CoT contains content.

Key metrics

• ASR (Attack Success Rate): Percentage of behaviours where the model complies rather than refuses.
• Full CoT ASR: ASR after considering truncated CoT responses heading toward compliance.
• Empty responses: Responses where the generation budget was exhausted before visible content was produced.

KL Divergence

Measures how much the abliterated variant’s output distribution diverges from the base model’s distribution on benign prompts.

Methodology

F.kl_div(logprobs_variant, logprobs_base, reduction="batchmean", log_target=True)

Full vocabulary first-token logits via model.generate(max_new_tokens=1, output_scores=True). Dataset: mlabonne/harmless_alpaca test[:100]. System prompt: “You are a helpful assistant.”

This matches the Heretic evaluator methodology, enabling fair comparison across techniques. KL divergence is hard to replicate exactly. CUDA version and GPU hardware can change results, so small variations are expected. We compare our KL values against existing card values where they exist to sanity check our setup, and a difference of 2 to 10 percent is normal and acceptable.

Interpretation scale

For the deep-dive on why the eval dataset, token depth, and thinking mode barely move the ranking, see Measuring KL Divergence for Abliteration Forensics .

Weight Forensics

All weight analysis uses Abliterlitics .

Analysis types

SVD (Singular Value Decomposition). Takes the difference between the original model’s weights and the abliterated model’s weights, then breaks that difference down into its fundamental components. Think of it like shining a light through a prism. The edit is white light, and SVD splits it into individual colours. If the edit is “rank-1,” it means the entire change points in a single direction. If it’s higher rank, the change is more complex and multi-directional.

Fingerprint. Compares every single weight tensor in the abliterated model against the original. Builds a complete map of exactly which weights changed and which stayed the same. This tells you how surgical or how broad the technique was. Did it touch 20 tensors or 200? Did it only modify output projections, or did it change attention weights, norms, and embeddings too?

Edit vector overlap. Compares the direction of the edits between two different techniques. If two techniques both modify the same weight tensor, are they pushing it in the same direction or completely different directions? Low overlap means they found different paths to the same result. High overlap means they’re doing essentially the same thing.

Per-layer analysis. Looks at which layers of the model carry the biggest edits. Some techniques concentrate changes in the middle layers, others at the top, others spread evenly. This matters because different layers handle different things. Early layers process basic patterns, middle layers handle reasoning, later layers control output behaviour.

Rank structure. Determines whether the edits live in a single direction or span multiple independent directions. A rank-1 edit is like drawing one line through the weight matrix. A rank-3 edit draws three lines. Lower rank generally means a more targeted, interpretable change.

Cross-technique alignment. Checks whether different abliteration methods found the same “refusal direction” or completely different ones. Across all our tests, the overlap is remarkably low. Different techniques consistently find structurally different solutions that produce identical behavioural outcomes. There is no universal abliteration subspace.

Key concepts

Tensors changed is how many weight matrices differ from the base model. Lower is generally more surgical.

Relative edit magnitude is how big the changes are relative to the original weights. Higher means more aggressive modification.

Cosine similarity measures whether two techniques’ edits point in the same direction. Low values mean they found different solutions.

Text-Encoder Forensics

Image and video generators that use an LLM as their text encoder get a different battery, measured at the conditioning interface rather than the chat pathway. The instruments:

KL divergence does not apply at this interface. There is no probability distribution to diverge, and the output head that would build one is not loaded in encoder deployments. The first results are in Abliterated Text Encoders, Measured: Z-Image Turbo and Qwen3-4B , with Krea 2, MiniMax H3, and LTX 2.5 to follow.

Verification layers

No single check is trusted on its own. Results pass through layered verification:

1. Deterministic parsing of raw JSON into the tables, no manual transcription
2. The LLM judge reads every safety response individually
3. Two independent reviewer models check every code change and every plan
4. The journal records what was measured, decided, and why, making every claim traceable
5. A human reads the final document before it publishes

Reproducibility

For recent and future comparisons we upload all tool artifacts to HuggingFace. That includes the raw logs and JSON output from every stage. We also push the repository branch in the exact state it was in when the comparison ran, so anyone can reproduce or verify the results. The text-encoder comparisons add their full artifact bundles: every generated image, every conditioning tensor dump, and the interactive report.

Hardware

Dual-GPU configurations use tensor parallelism with TP=2 and BitsAndBytes 4-bit or BF16 with CPU offloading depending on model size.

Tools

• Abliterlitics : our open-source analysis toolkit
• HarmBench : safety evaluation framework
• lm-evaluation-harness : capability benchmarks
• ComfyUI : generation backend for text-encoder forensics