SimpleLoRIF Filters Safety-Degrading Training Data

Researchers introduced SimpleLoRIF, a method for finding instruction-tuning examples that weaken language-model safety even when the examples appear harmless. The paper was published on October 27 and submitted to Hugging Face by author Pratyush Maini on November 2. SimpleLoRIF compares gradient embeddings (numerical representations of how training examples change a model) with unsafe behavior, then filters examples whose gradients align with that behavior. By simplifying a low-rank attribution method and limiting backpropagation to the final four transformer blocks, the researchers say it can process datasets with about a million examples.

The team first tested the method on emergent misalignment, where insecure-code training data was linked to broadly harmful behavior after fine-tuning. In misaligned models, gradient-space detection reached an AUC as high as 0.865. The researchers then created SafeTulu, a version of Tulu 3 with the 20% of examples most aligned with unsafe responses removed. Training on SafeTulu reduced GCG attack success rates by up to 22% across Qwen, Llama and Olmo, outperforming LLM-as-a-judge and earlier gradient-based filtering while largely preserving or improving capability.

Analysis of removed examples pointed to tool-execution failures and narrative arithmetic as possible under-studied sources of safety degradation. The researchers released SafeTulu and a Hugging Face Space, where researchers can create customized versions by changing the removal fraction or filtering selected subsets.

Researchers introduced SimpleLoRIF, a method for finding instruction-tuning examples that weaken language-model safety even when the examples appear harmless. The paper was published on October 27 and submitted to Hugging Face by author Pratyush Maini on November 2. SimpleLoRIF compares gradient embeddings (numerical representations of how training examples change a model) with unsafe behavior, then filters examples whose gradients align with that behavior. By simplifying a low-rank attribution method and limiting backpropagation to the final four transformer blocks, the researchers say it can process datasets with about a million examples.

The team first tested the method on emergent misalignment, where insecure-code training data was linked to broadly harmful behavior after fine-tuning. In misaligned models, gradient-space detection reached an AUC as high as 0.865. The researchers then created SafeTulu, a version of Tulu 3 with the 20% of examples most aligned with unsafe responses removed. Training on SafeTulu reduced GCG attack success rates by up to 22% across Qwen, Llama and Olmo, outperforming LLM-as-a-judge and earlier gradient-based filtering while largely preserving or improving capability.

Analysis of removed examples pointed to tool-execution failures and narrative arithmetic as possible under-studied sources of safety degradation. The researchers released SafeTulu and a Hugging Face Space, where researchers can create customized versions by changing the removal fraction or filtering selected subsets.