I’d use deep k-NN to flag samples for review - not to decide what to delete. It detected over 99% of injected poisons in reported CIFAR-10 tests, but that does not guarantee the same results on your data.[1]
The check is simple: does a sample’s label disagree with the labels of nearby samples in embedding space? Correctly labelled poisons can fail this check. So can clean samples from small classes or class boundaries.
Before excluding data, I’d check:
- Detection and false flags: measure poison recall, precision and clean samples flagged, including results for each class.
- Settings and cost: test the embedding model, neighbourhood size (k), class balance and search method.
- Training results: confirm that attack success falls without unacceptable losses in clean accuracy.
The trade-off matters: one later study reported 20% precision - only one in five flagged samples was poison.[2] My rule: quarantine first, review before removal, and repeat the checks when your data or model changes.
Lecture 4: Training Data Poisoning
Evidence from Clean-Label Poisoning Tests
Peri et al.’s Deep k-NN Defense Against Clean-label Data Poisoning Attacks reports detecting more than 99% of injected poisons across feature-collision and convex-polytope attacks.[1] The method flags samples when their labels disagree with the vote among nearby samples in feature space.
Check the primary paper for the exact model, neighbour layer and training setup.
Poison Detection and Clean Accuracy
The detection rate counts injected poison examples, not attack attempts or all flagged records. Feature-collision attacks place poisons close to a target in representation space. Convex-polytope attacks use several correctly labelled poisons whose feature vectors surround or approximate the target representation.
These results come from one CIFAR-10 benchmark. To judge the detection rate, you also need to know the cost to clean data.
| Attack type | Reported detection result | Clean-performance effect | Experimental limitation |
|---|---|---|---|
| Feature collision | Over 99% of injected poisons detected.[1] | Poisons were removed without compromising overall model performance.[1] | The available text does not report the exact benign-removal rate or change in clean-test accuracy. |
| Convex polytope | Over 99% of injected poisons detected.[1] | More false positives than feature collision led to more benign examples being removed, potentially reducing test accuracy.[1] | The available text does not report exact benign-removal or post-defence attack-success rates. |
| Other or adaptive attacks | Not reported | Not reported | Results should not be extended beyond the tested attack families. |
How k and Class Balance Affect Results
The paper reports 100% defence success once the normalised k ratio exceeded 0.2.[1] However, the available summary does not define the ratio’s denominator or give matching benign-removal figures. It also mentions class-balance tests but supplies neither class proportions nor numerical outcomes.
These settings affect which label–embedding disagreements trigger flags, so treat the k threshold and class-balance findings as specific to the tested configuration. Test several k values, measuring poison recall and benign-removal rates separately for each class, rather than adopting one setting for every case. The settings control how many samples are quarantined; they do not establish that the method is safe in every case.
Assumptions and Limits of Deep k-NN
Those detection rates assume that nearby embeddings usually share labels. Deep k-NN flags label–embedding mismatch. It does not prove that a sample is poisoned.
The original defence freezes representations before victim-model training and uses Euclidean distance.[1] Its reliability depends on the encoder, normalisation and reference pool. Missing classes, duplicates or contamination can skew neighbour votes. Early layers may also track texture rather than features that separate classes.
These assumptions can fail at class boundaries and for under-represented groups.
False Flags in Minority Classes and Boundary Samples
Small classes have fewer same-class neighbours, making benign samples easier to outvote. Boundary samples, natural outliers and weak representations can cause the same disagreement.
Report benign flags for each class or subgroup as both counts and percentages, alongside class-specific retention and downstream accuracy. Inspect unusual samples before deleting them.
The same neighbourhood rule can also miss poisons that blend into their assigned class.
How Attacks Can Evade Filtering
A later scalable clean-label poisoning study reported that increasing the neighbourhood size eliminated its attack. But detection precision fell to 20%: only one in five flagged samples was a poison.[2]
| What it detects | Why it helps | What must hold | Where it breaks |
|---|---|---|---|
| Label–embedding mismatch | Finds locally inconsistent samples | Nearby embeddings have compatible labels | Benign boundary samples and outliers trigger flags |
| Larger neighbourhoods | Makes small poison groups less influential | Broader neighbourhoods remain class-consistent | More clean samples may be removed |
| Fixed embeddings | Makes pre-training checks reproducible | The encoder reflects relevant class distinctions | Weak or mismatched features produce misleading votes |
Pair neighbour filtering with provenance checks, duplicate detection and post-training evaluation. Keep flagged records for review; do not auto-delete them.
Validate Deep k-NN in Your Training Pipeline
::: @figure 
{Deep k-NN Validation: Flag, Quarantine, Review}
:::
Before deployment, test deep k-NN on your own data and embeddings. Check whether label–embedding disagreement separates poisons from benign samples. Define the dataset, label taxonomy, class balance, preprocessing, model and embedding layer used for neighbour comparisons. Use the same checkpoint and representation as training or inference; otherwise, the detector measures a different geometry.
Set aside clean reference data, a held-out validation set and, where possible, controlled clean-label poisons. Record data sources, transformations, label revisions, versions and access history so the tests can be reproduced.[3][4]
With the validation setup fixed, test how well the detector flags poisons without harming clean data.
Measure Detection, Accuracy and Cost
Compare neighbourhood sizes such as k ∈ {1, 3, 5, 10, 20}, along with class-scaled values. Select k and the flagging threshold using tuning data, not the final test set.
For each k, report poison recall, precision, false-positive rate, benign removal rate, retained volume, clean validation accuracy, macro-F1, per-class recall and post-defence attack success rate. Where possible, repeat tests across seeds or resampled reference sets. Compare all-data, removal, quarantine and no-defence conditions on a consistent basis.[3][5]
Measure embedding extraction, index building and neighbour-search time separately. Also record peak memory, storage and retraining or review latency. Compare exact and approximate search on a representative subset, recording the percentage of filtering decisions that change, not just neighbour overlap. Inspect changes near the threshold before accepting a faster configuration.
Quarantine Flags and Recheck Updates
In production, make flags reversible so false positives and boundary cases can be reviewed. Keep the original record, label, model version, neighbour IDs, distances, vote share, threshold, timestamp and reviewer decision. Define who can quarantine and release samples, and restrict access to data changes and review records.
Re-run evaluation after label revisions or changes to class balance, preprocessing, embeddings, model architecture, training data, the index or approximate-search settings. Document intended use, known limitations, acceptance thresholds and residual risk.[3][5]
Conclusion: Screening, Not a Guarantee
The evidence supports deep k-NN as a screening gate, not a standalone defence. Label–embedding disagreement points to a local mismatch; it does not prove poisoning. The reported CIFAR-10 results support its use as a baseline for the attacks tested, rather than as a general defence.[1]
Reliability depends on representation quality, k, class balance and attacker behaviour. Minority-class and boundary samples can produce costly false flags, while later studies report evasion in some settings.[6][7] What matters for your pipeline is whether the flags help and whether the false-positive rate is acceptable.
Base exclusion on a pilot. Before excluding any sample, set thresholds for benign removal, clean-accuracy loss and residual attack success. Exclude data only when attack success drops enough to meet your target, clean accuracy stays within tolerance and benign removals remain within the agreed threshold. Otherwise, keep flagged samples in quarantine for review.
FAQs
How can I distinguish poisoned samples from legitimate outliers?
Reconstruction scores and labels alone aren’t enough to reliably tell poisoned samples apart from rare but valid outliers. Use several checks to prioritise manual review: look for discrepancies in provenance, metadata and labels; check for unusual covariance patterns in feature space; and test whether predictions change under small, label-preserving perturbations.
Before making permanent changes, retrain models both with and without the suspected records. Compare their trigger behaviour and performance on clean data.
How should I choose k for an imbalanced dataset?
Don’t rely on a fixed k, such as k=2: it can force splits even in clean data. Vary k and check whether the clusters stay stable [1].
Small classes can produce unstable directions and unreliable rankings, so splits alone don’t prove poisoning. Look for patterns that persist across k settings. Then check those patterns against sample provenance, timestamps, or trigger-like features before considering exclusions [1].
How can I validate deep k-NN without known poisons?
Treat deep k-NN as a diagnostic tool, not proof of integrity. Set historical baselines and use unsupervised anomaly detection to flag unusual embeddings. Set thresholds using verified-clean validation data, based on either empirical percentiles or the median plus k times the median absolute deviation.
Place flagged samples in reversible quarantine for documented expert review. Don’t delete them automatically: doing so risks losing legitimate rare data and biasing models.