Can AI Score a Sleep Study?

The Question Sounds Simple. It Isn't.

Someone reads your overnight sleep study and decides, thirty seconds at a time, what stage of sleep you were in. That's the job. It takes hours. It's tedious. It costs money.

So of course people asked whether software could do it. And the answer is yes, mostly, and it's already happening in real sleep labs.

But there's a trap hiding in every headline that says an algorithm "matches human experts." Human experts don't match each other. Two well-trained scorers reading the same night will disagree on roughly one epoch in six. So when a company says its model hits 85% agreement with expert scoring, you have to ask: agreement with which expert?

This post is about that ceiling. What scoring is, how much humans disagree, why that caps what any accuracy number can mean, and where automation actually earns its keep.

Key Facts

• Sleep is scored in 30-second chunks called epochs, using brain, eye, and chin muscle signals (StatPearls)
• Overall agreement between scorers on sleep stages: 82.6%, across more than 2,500 scorers (Rosenberg & Van Hout, JCSM 2013)
• Agreement on N1, the lightest stage, is just 63.0%. Deep sleep (N3) is 67.4% (JCSM 2013)
• Breathing events are messier still: scorers agreed on central apneas only 52.4% of the time (Rosenberg & Van Hout, JCSM 2014)
• A deep-learning system trained on 15,660 people scored "as accurately as the best of the human experts" (Perslev et al., npj Digital Medicine 2021)
• The AASM's position: AI must be a partner to, not a replacement for, human oversight (AASM position statement, 2025)

What "Scoring a Sleep Study" Actually Means

A polysomnogram — the full in-lab sleep study, usually shortened to PSG — records a lot of channels at once. Brain waves from several scalp electrodes. Eye movements. Chin and leg muscle activity. Heart rhythm. Airflow at the nose and mouth. Chest and belly effort. Blood oxygen. Sometimes audio and video [^1].

Then a trained technologist chops the whole night into 30-second slices called epochs and labels each one [^1]. Eight hours of sleep is roughly 960 epochs. For each slice, they decide: awake, N1, N2, N3, or REM. Then they go back through and mark every apnea, hypopnea, limb movement, and arousal.

Sleep staging leans on three signal types: EEG (brain waves), EOG (eye movements), and submental EMG (chin muscle tone) [^1]. That combination is the whole basis of the stage labels. No EEG, no staging. That matters later.

If you want the map of what all those numbers mean once they land on the report, we wrote a full translation of a sleep doctor's report.

Two Experts, One Night, Different Answers

Here's the part almost nobody outside sleep medicine knows.

The American Academy of Sleep Medicine (AASM) runs an inter-scorer reliability program. Scorers all over the country label the same record fragments, and the program tracks how often they land on the same answer. In the published analysis, more than 2,500 scorers — most with three or more years of experience — worked through 1,800 epochs and produced over 3.2 million individual scoring decisions [^2].

Overall agreement on sleep stage: 82.6% [^2].

Read that again. Roughly one epoch in six, two qualified professionals looking at identical data write down different answers.

And the average hides the worst of it. Broken out by stage [^2]:

• REM had the highest agreement
• Wake and N2 came close behind
• N3 (deep sleep): 67.4%
• N1 (lightest sleep): 63.0%, the lowest of all

N1 is the flickery in-between state where you're drifting off. It's short, it's transitional, and it's where scorers argue most. The paper also found scorers struggled specifically at the boundaries — the last wake epoch before sleep onset, the first N2 after N1, the first REM after N2 [^2]. Transitions are hard because sleep doesn't actually change on a 30-second grid. We just score it that way.

Breathing Events Are Even Messier

Staging isn't the only judgment call. The same program looked at respiratory event scoring with more than 3,600 scorers [^3].

The headline number looks great: 93.9% overall agreement. But that's inflated, because most epochs contain no event at all, and everyone agrees on nothing-happened epochs 97.4% of the time [^3].

When something did happen, agreement fell apart [^3]:

• Obstructive apnea: 77.1%
• Hypopnea: 65.4%
• Central apnea: 52.4%

Central apnea — a pause where the brain simply doesn't send the breathe signal — got a coin-flip. The most common disagreement on obstructive apneas was scorers calling them hypopneas instead [^3].

This is the real reference standard. Not truth. A noisy, human, somewhat inconsistent set of labels that we agreed to treat as truth because it's the best we have.

Why That Caps What "AI Accuracy" Can Mean

Now the logic gets uncomfortable.

Every automated sleep-scoring model is trained on human labels and tested against human labels. That's the only ground truth available. Which means the ceiling on measured accuracy isn't 100% — it's roughly wherever humans agree with each other.

If two experts agree 82.6% of the time, and an algorithm agrees with one expert 85% of the time, the algorithm has not "beaten" humans in any meaningful way. It has learned to sit near the middle of a cloud of human opinion. Sometimes it's closer to the consensus than any single scorer, which is a real advantage. But it is still being measured against a moving target.

"Matches human experts" means matching a noisy reference, not matching truth. Anyone selling you an accuracy percentage without telling you what the reference was is telling you almost nothing. We made the same argument about consumer devices in how to read sleep tracker accuracy claims.

There's a second wrinkle. Models trained on one lab's scorers inherit that lab's habits. Where does one scorer draw the N1/N2 line? Do they use the 3% or 4% desaturation rule for hypopneas? The algorithm learns those conventions along with the physiology.

So How Good Is Automated Scoring, Really?

Good. Actually good.

U-Sleep, published in npj Digital Medicine in 2021, is a publicly available deep-learning staging system trained and evaluated on PSG recordings from 15,660 participants across 16 clinical studies ¹. It works on arbitrary combinations of typical EEG and EOG channels, and it holds up on patient groups and recording protocols that weren't in its training data.

The authors compared it against consensus scores from a clinic it had never seen. Their finding: U-Sleep "performs as accurately as the best of the human experts" ¹.

That's a legitimate result, and it points at where automation actually helps:

• Consistency. A model applies the same rule at 2 a.m. as at 2 p.m., on epoch 12 and epoch 900. Humans get tired.
• Speed and cost. Hours of technologist time per study is a real bottleneck in sleep medicine.
• Finer resolution. U-Sleep can label at intervals shorter than 30 seconds, which the authors argue reveals structure the standard grid hides ¹.
• Access. In places without a trained scorer, an algorithm beats nothing.

What About Large Language Models?

Different question. Chat-style models reading EEG is a much newer and shakier idea.

A 2026 preprint — not yet peer reviewed — introduced BrainBench, a benchmark for testing large language models on EEG tasks. It's built from 17 datasets across four areas, including a sleep assessment section with 43 tasks covering sleep architecture, staging, spectral interpretation, and event detection ².

Results: performance varied a lot across models, task difficulty, and how the model was allowed to work. The best overall score stayed below 80 ².

So: interesting research direction, nowhere near a clinical tool. Treat any claim that a chatbot can read your sleep study as marketing until a peer-reviewed version says otherwise.

Agreement, Side by Side

Scan that table before you believe any accuracy claim. The bar being cleared is lower than it sounds.

What the Sleep Field Actually Says

The AASM has taken a position, twice.

In 2020, its position statement said the goal of AI integration "should be to augment, not replace, expert evaluation of sleep data," and that "full physician review is expected after algorithm-based scoring" ⁵.

The updated 2025 statement holds the same line: AI "must be a partner to, and not a replacement for, clinician (ie, human) oversight," and responsible use requires attention to validation, ongoing accuracy after deployment, safety, and transparency ⁶.

That's the honest summary. Software does the first pass. A human checks it. A physician signs it.

Where Komori Fits

Nowhere in this workflow, and we should say that plainly.

Komori is a wellness device, not a diagnostic one. It has no EEG. No electrodes, no wires, nothing touching you. Without brain, eye, and chin signals, there is no sleep staging in the clinical sense — that isn't a software problem, it's a physics one ⁷.

What Komori is being built to do is different work: a contactless device that senses from above the bed, paired with a base unit on the nightstand for power, Wi-Fi, and local processing. When it ships, it's designed to track sleep position, movement and restlessness, bed-exit events, sound events, and room conditions like temperature, humidity, CO2, light, and noise — every night, in your own bed.

That's a different axis than a sleep study. A PSG gives you one night of very deep detail. Long-run home data gives you shallow detail across hundreds of nights. Neither replaces the other, and we've written about why we refuse to squash all of it into a single score.

The interesting part of the scoring story isn't that machines are catching up to people. It's that the "gold standard" was never gold. It was a careful, imperfect human judgment we all agreed to trust.

If you have a sleep study in hand, the person to talk to about it is the sleep specialist who ordered it. They know what the numbers mean, and they know how much to trust them.

---

Footnotes

1. Perslev M, Darkner S, Kempfner L, Nikolic M, Jennum PJ, Igel C. "U-Sleep: resilient high-frequency sleep staging." npj Digital Medicine, 2021. ↩ ↩² ↩³

2. Zhou Y, Zhao S, Chen Y, Wu C, Wang J, Li S, Pan G. "BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding." arXiv preprint 2608.04156, 2026. Preprint — not peer reviewed. ↩ ↩²

3. Rosenberg RS, Van Hout S. "The American Academy of Sleep Medicine inter-scorer reliability program: sleep stage scoring." Journal of Clinical Sleep Medicine, 2013;9(1):81–87. ↩ ↩² ↩³ ↩⁴

4. Rosenberg RS, Van Hout S. "The American Academy of Sleep Medicine inter-scorer reliability program: respiratory events." Journal of Clinical Sleep Medicine, 2014;10(4):447–454. ↩ ↩² ↩³ ↩⁴ ↩⁵

5. Goldstein CA, et al. "Artificial intelligence in sleep medicine: an American Academy of Sleep Medicine position statement." Journal of Clinical Sleep Medicine, 2020;16(4):605–607. ↩

6. Oks M, et al. "Artificial intelligence in sleep medicine: an updated American Academy of Sleep Medicine position statement." Journal of Clinical Sleep Medicine, 2025;21(11):1953–1955. ↩

7. Sankari A, Slowik JM. "Sleep Study." StatPearls Publishing, 2026. ↩