when a sleep-scoring classifier fails silently

Automated sleep scoring is now standard in rodent sleep research: it is faster than manual scoring and, under ordinary conditions, close enough to it. The assumption underneath that practice is that classifier accuracy is roughly stable across experimental conditions.

This work asked whether that holds when the condition is a drug intended to change sleep architecture — the exact situation in which the automated scores matter most.

Somnivore is sleep-scoring software: a human labels a subset of EEG/EMG epochs, the system classifies the full recording into Wake, NREM, and REM at four-second intervals, and a human then reviews and corrects the automated output. Both the raw automated scores and the corrected scores are retained, which makes the pair a usable ground-truth comparison.

The study behind this analysis was an apamin dose-response experiment in mice, run in Stanford's Heller Laboratory.

Somnivore's EDF+ output does not deserialize cleanly with off-the-shelf readers, so the first piece was a custom parser for its annotation layout. The second was alignment: automated and corrected score streams must be matched epoch-for-epoch across 47 recordings, with mismatched lengths and offsets resolved rather than truncated away.

On top of the aligned streams, every missed REM epoch was classified by what it was instead assigned to, giving a per-condition confusion structure rather than a single accuracy number. A linear mixed-effects model handled the repeated-measures design (multiple doses per animal), and the 5 ng post-injection period was subdivided into four-hour windows to see whether the effect was uniform in time.

One cohort was identified and excluded after an anomaly in the aligned scores was traced back to an artifact in the source recordings rather than to the analysis.

From saline through 4 ng, the classifier looked reliable. At 5 ng it did not.

The interesting part is not that accuracy fell. It is that REM prevalence in the corrected scores stayed approximately unchanged across the same transition. The animals were still entering REM at similar rates; the classifier had stopped recognizing it.

The error analysis showed the missed epochs were increasingly assigned to Wake rather than NREM — a directional failure, not diffuse noise. An aggregate REM-time summary computed from raw automated scores would have reported a REM deficit that the corrected data does not support.

At some doses, using the raw automated scores would have changed the interpretation of the experiment. The biological phenomenon had not disappeared; the measurement system had changed behavior — and the summary statistic most people would look at was capable of hiding that.

What is demonstrated here is the failure and its direction. Why the classifier fails at this dose is not established: apamin-associated changes in EEG/EMG features are a plausible mechanism, not a result.

More animals to tighten the dose-response estimate, finer time bins within the post-injection window, and spectral analysis of the EEG/EMG signal to test whether the feature space the classifier relies on is what moves.