A random split scores a stress classifier perfect on data with nothing to learn

Wearable stress detectors are often scored by cutting recordings into overlapping windows and splitting the windows at random into training and test sets. We tested what that split measures. On planted data built on the same windows and timing as the WESAD recordings, where each person and each condition block carries its own random offsets and nothing carries from one person to the next, a random split over windows scored an AUC of 1.000 and a held-out-person split scored 0.469.c1 On the real chest recordings of 15 people (2,122 windows of 60 s, 30 percent of them stress), the random forest held across people at a held-out-person AUC of 0.966, against 1.000 under the random split.c2 The random split inflated the AUC by 0.033 for the random forest and 0.086 for logistic regression.c3 ECG, EDA and skin temperature together reached 0.97 for a held-out person, and acceleration with EMG reached 0.79.c4 Across the 15 people, the rise in reported anxiety did not follow the size of the bodily change: Spearman rho 0.03 with EDA, 0.02 with heart rate and -0.45 with breathing rate (p 0.09).c5 The first run failed its own planted check because we set its bar at 0.90, within 0.012 of the best score that planted shift allows; a second preregistered run repaired the plant and passed.

WESAD, the Wearable Stress and Affect Detection dataset of Schmidt and colleagues [1], records 15 people with a chest sensor and a wrist sensor during a baseline, amusing film clips and the Trier Social Stress TestDefinition: Trier Social Stress Test psychology and psychoneuroendocrinology A laboratory stressor in which a person prepares and gives a short public talk and then does mental arithmetic aloud in front of an evaluating panel. Excludes daily stress, chronic stress and trauma. full definition, in which a person gives a short public talk and then does mental arithmetic aloud before an evaluating panel. The dataset paper reports classifiers scored with every person held out in turn, and WESAD has since become a common benchmark.

Many later studies score their models with a random split over windowsDefinition: random split over windows machine learning evaluation; called record-wise cross-validation in the clinical literature Assigning windows to training and test sets at random, so that windows from the same person and the same recording block fall on both sides. Excludes any split that keeps all of one person's data on one side. full definition: the windows are assigned to training and test sets at random, so windows from the same person and the same recording block fall on both sides. The windows are usually overlapping windowsDefinition: overlapping windows signal processing Fixed-length stretches cut from a recording at a step shorter than their length, so that neighbouring windows share samples. Here 60 s windows every 15 s share 75 percent of their samples. Excludes windows cut end to end with no shared samples. full definition, fixed-length stretches cut at a step shorter than their length. Here a 60 s window is cut every 15 s, so neighbours share 75 percent of their samples.

The clinical machine-learning literature has argued over this choice. Saeb and colleagues [2] showed on wearable and phone sensor data that record-wise and subject-wise cross-validation can give very different error rates, and traced the gap to the classifier learning to recognise people. Little and colleagues [3] replied that subject-wise splits can also mislead in some settings. Chaibub Neto and colleagues [4] built a permutation test for identity confounding and, on mobile health data, found strong evidence of it under record-wise splits. Our searches on 9 November 2026 also returned recent WESAD studies and open audits that make the same point about random splits; we read those at the level of search summaries and do not cite their numbers.

This paper adds a known answer. We build data in which nothing can carry from one person to another, run it through the same windows, features and models as the real data, and measure what the random split reports. Then we ask three questions of the real data: whether the classifier holds for a person it has not seen, which sensors carry the signal, and whether the size of a person's bodily response follows the size of the stress they report. On the last question, Campbell and Ehlert [5] reviewed acute psychosocial stress studies and asked whether emotional and physiological responses correspond.

The design was written as a preregistrationDefinition: preregistration research method (metascience) A dated written plan for a study, fixed and hash-locked before any result is seen: the question, the data, the statistic, the controls, the forecasts and the kill conditions. Excludes a plan revised after looking at the outcome, and a protocol written only for the record after the run. full definition, a dated plan fixed and hash-locked before any result is seen, on 31 October 2026 at 22:37 UTC. Window features were extracted and fixed before any label was compared with them. The decision it informs was stated in the file: how much to discount a wearable stress classifier reported under a within-person random split, and whether its label reads stress or a protocol.

The locked file names the data, the windows and features, four splits, two models, the sensor groups, the felt-stress test, a planted checkDefinition: planted check research method; called a positive control in experimental science Data built so that the right answer is known, run through the same analysis before the real data, to show that the analysis can see the effect when it is there. Excludes any check made on the real data, which cannot show what the analysis misses. full definition (data built so the right answer is known, run before the real data), a controlDefinition: control statistics and experimental design A version of the data with the structure under test removed (for example shuffled or pooled), scored the same way as the real data, so the real score has something to beat. Excludes a comparison with a different dataset or with a published number. full definition (a version of the data with the structure under test removed), forecasts and kill conditionsDefinition: kill condition research method (our usage, after Popper's falsification) A rule written in the preregistration that, when met, decides against the claim with no room for judgement after the run. Excludes reasons found after the run for setting a result aside. full definition, rules that decide against the claim with no room for judgement after the run.

Kill conditions, without discretion:

The first run met the second part of the first rule: the planted shift scored 0.824 against a bar of 0.90. By its own rule that run was voidDefinition: void research method (our usage) The verdict of a run whose planted check failed or whose sample fell below its stated minimum. It says nothing for or against the claim. Excludes a null result, which comes from a working instrument. full definition, a verdict that says nothing for or against the claim. A calculation after the run showed the bar left almost no room: for the planted shift of 2 the best possible AUC is about 0.91 (section Methods). A second preregistration, locked at 22:39 UTC the same evening, kept the real-data handling unchanged, tested planted shifts of 2, 3 and 4 on a fresh seed, and required the shift of 4 to reach 0.90 for a held-out person. If it passed, the first run's real-data results were to be credited. It passed. The second plan was written after the first run's results were seen, and its forecasts for the real data copy the first run's values; we report it as a repair, not as a blind test.

The data are WESAD [1], obtained from the authors' page at the University of Siegen; the UCI repository record (doi:10.24432/C57K5T) points to that page. The terms allow scientific, non-commercial use with credit to the dataset's owners. This paper reports statistics only. Subjects are pseudonymous (S2 to S17; there is no S1 or S12); no attempt was made to identify anyone.

We used the chest device only, recorded at 700 Hz, read from the text files: ECG, EDA, EMG, skin temperature, three-axis acceleration and respiration. The pickled files were not opened. Condition times and the questionnaire answers came from each person's questionnaire file. The task was the binary one of the dataset paper: stress (the Trier test) against baseline and amusement. Windows of 60 s were cut every 15 s inside each condition. Windows with any missing feature were dropped, leaving 2,122 windows from 15 people, 30 percent of them stress.

Electrodermal activity (EDA)Definition: electrodermal activity (EDA) psychophysiology The electrical conductance of the skin, which rises with sweat gland activity under sympathetic nervous control. Excludes a direct reading of emotion; temperature and movement also change it. full definition, used throughout, is the electrical conductance of the skin, which rises with sweat gland activity under sympathetic control.

Twenty features in six groups were computed per window. ECG: heart rate, the standard deviation of beat intervals, the root mean square of successive interval differences, and the ratio of the two as a rough balance measure. EDA: mean, standard deviation, slope and the count of skin conductance responses. Temperature: mean and slope. Respiration: breath count, amplitude and standard deviation. Acceleration: mean magnitude, its standard deviation and the mean of each axis. EMG: root mean square and 90th percentile of the rectified signal.

Two classifiers were scored. A random forestDefinition: random forest machine learning A classifier that averages many decision trees, each grown on a resampled copy of the training data with a random subset of features tried at each split. Here 100 trees with at least 3 windows per leaf. Excludes any model tuned on the test person. full definition averages 100 decision trees grown on resampled data with at least 3 windows per leaf. Logistic regressionDefinition: logistic regression statistics A classifier that models the log odds of the positive class as a weighted sum of the features. Here on standardised features with an L2 (ridge) penalty at strength C = 1. Excludes interactions between features, which it does not model unless they are added by hand. full definition models the log odds of stress as a weighted sum of standardised features. The score is the AUCDefinition: AUC (area under the ROC curve) statistics and machine learning The probability that a classifier gives a randomly chosen positive case a higher score than a randomly chosen negative case. 0.5 is chance and 1 is perfect separation. Excludes accuracy at one threshold, and any statement about how well the scores are calibrated as probabilities. full definition, the probability that a randomly chosen stress window gets a higher score than a randomly chosen non-stress window. With ranks $r_i$ of the scores, $n_1$ stress windows and $n_0$ others,

where $R_1$ is the sum of the stress windows' ranks. AUC is reported two ways: global, over all held-out windows pooled, and within-person AUCDefinition: within-person AUC machine learning evaluation, our usage The AUC computed on one held-out person's windows alone, then averaged over people. It asks whether the classifier ranks that person's stress windows above their other windows. Excludes the global AUC, which pools all held-out windows and also rewards separating people from each other. full definition, computed on each person's windows and then averaged over people.

The leakage gap is the difference between the random-split and held-out-person global AUCs:

The planted data keep the real windows' people, conditions and timing and replace the features. For window $w$ of person $s$ in condition block $c$, feature $f$ is

where ${ϵ}_{wf}$ and $u_{scf}$ are independent standard normal draws, the second shared by every window of one person's block, $y_w$ is 1 for a stress window and 0 otherwise, and $d$ is a planted shared shiftDefinition: planted shared shift research method, our usage A fixed amount added to one feature in every stress window of every person in planted data, in the same direction for all, so that a classifier tested on a new person has something real to find. Excludes the per-person, per-block offsets, which point in a random direction and carry nothing from one person to the next. full definition added to the first feature only. With $d=0$ each block has its own offsets in a random direction and nothing carries from one person to the next; a held-out person should score near 0.5. With $d>0$ a held-out person has something real to find.

For one feature, the stress and non-stress values each have variance $1+{0.3}^2=1.09$, so the ceiling AUC for a shiftDefinition: ceiling AUC for a shift statistics, our derivation The AUC a perfect reader of the one shifted feature could reach when the stress and non-stress values of that feature are normal with a variance of 1.09 each (a block offset of variance 1 plus window noise of variance 0.09). Excludes the effect of the 19 features that carry no shift, and of the small number of people. full definition is

where $N$ is the standard normal cumulative distribution. At $d=2$ this gives 0.912, which is why a bar of 0.90 left no room.

Each sensor group, physiology without motion or breath (ECG, EDA and temperature) and motion alone (acceleration and EMG) were scored by held-out-person within-person AUC with the random forest. Whether each of the six single groups beats chance was tested with logistic regression against 100 block permutations, in which each person's stress label is moved to one of their condition blocks chosen at random, so block structure and the share of stress windows stay. The p value counts permutations at or above the real score, with one added to both counts:

The six p values were adjusted with the Benjamini-Hochberg correctionDefinition: Benjamini-Hochberg correction statistics, multiple testing A rule that adjusts a set of p values so that, among the tests called significant, the expected share of false discoveries stays at or below a chosen level q. Excludes control of the chance of even one false discovery, which is a stricter rule. full definition, which keeps the expected share of false discoveries among the tests called significant at or below q 0.05. For p values sorted in increasing order, the $i$th of $m$ is scaled to $m{p}_{\left(i\right)}/i$, and each adjusted value is the smallest scaled value at or after its place in the order.

For each person, the change in the six-item STAIDefinition: STAI, six-item form psychology A short form of the State-Trait Anxiety Inventory in which a person rates six statements about how they feel at that moment. Positive items are reverse scored so that a higher total means more anxiety. Excludes a clinical diagnosis, and anxiety as a lasting trait. full definition total, a short state-anxiety questionnaire in which positive items are reverse scored, was taken from baseline to stress. The same person's change in mean EDA, heart rate and breath count from baseline windows to stress windows was compared with it across the 15 people by Spearman rhoDefinition: Spearman rho statistics The correlation between the ranks of two quantities. It runs from -1 to 1 and measures how well one quantity's order follows the other's, whatever the shape of the relation. Excludes the size of the change, and any causal direction. full definition, the correlation of ranks.

The second run reports a 95% intervalDefinition: 95% interval statistics The range that holds the middle 95 percent of the bootstrap or control draws for a number. We report it beside the number. Excludes a statement that the true value lies inside with 95 percent probability. full definition on the mean within-person AUC by bootstrap by personDefinition: bootstrap by person statistics, resampling Redrawing people with replacement, 2,000 times, and recomputing the mean of the per-person scores each time. The middle 95 percent of those means is the interval. Excludes redrawing single windows, which would treat one person's many windows as many independent people. full definition: people are redrawn with replacement 2,000 times and the mean is recomputed each time.

Planted data with nothing to learn ($d=0$) were run in both runs. The random split read 1.000 (first run, 0.99999 before rounding) and 1.000 (second run). The held-out-person split read 0.469 and 0.391. Both passed their rules.

The planted shift failed in the first run, at 0.824 against the bar of 0.90 for $d=2$. In the second run, on a fresh seed, the forest reached 0.692 at $d=2$ (ceiling 0.912), 0.987 at $d=3$ (ceiling 0.979) and 0.994 at $d=4$ (ceiling 0.997). The rule on the shift of 4 passed. The value at $d=3$ sits 0.008 above its computed ceiling; the ceiling counts one feature and two normal distributions, and we did not trace the difference. The two runs at $d=2$ differ by 0.13, so this planted check is noisy at small shifts.

The block-shuffle control moves each person's stress label onto one of their own condition blocks at random and scores logistic regression on held-out people. Over 100 draws in the first run it gave a mean of 0.487 and a 95th percentile of 0.656 (p from drawsDefinition: p value (from draws) statistics The share of control draws scoring at or above the real score, with one added to the count and to the number of draws, so it is never zero. Excludes a p value from a textbook formula that assumes a distribution. full definition 0.0099); over 40 draws in the second it gave 0.472 and 0.594 (p 0.024). The real logistic score of 0.896 clears both. This control's 95th percentile marks the top of its null distributionDefinition: null distribution statistics The spread of scores a control gives over many random draws. Its 95th percentile is the bar a real score must clear. Excludes a theoretical distribution assumed rather than drawn from the data. full definition, the spread of scores the control gives over random draws.

A random split over windows scored an AUC of 1.000 and a held-out-person split scored 0.469.c1 That is the planted data with nothing to learn, in the first run. The random forest separated stress from non-stress windows perfectly by recognising each block's offsets in windows it had already seen. Figure 1 sets these planted values beside the real ones.

Table 1 gives both models under the four splits. The random forest held across people at a held-out-person AUC of 0.966, against 1.000 under the random split.c2 The second run, on a new seed, gave 0.999 and 0.966. The random split inflated the AUC by 0.033 for the random forest and 0.086 for logistic regression.c3 Scaling each person to their own baseline brought the logistic score from 0.896 to 0.971 and left the forest at 0.964.

The mean within-person AUC for a held-out person was 0.979, with a 95% interval of 0.961 to 0.994 by bootstrap over people. The worst person under the forest read 0.912 (S2); under logistic regression the worst read 0.646 (S15). Figure 2 shows every person and the planted-shift check.

By sensor group, held-out person, random forest, the within-person AUCs were ECG 0.92, EDA 0.89, respiration 0.85, acceleration 0.82, temperature 0.60 and EMG 0.45 (Figure 3). In the logistic permutation test five groups passed the correction at q 0.012 each; EMG did not (q 0.33). ECG, EDA and skin temperature together reached 0.97 for a held-out person, and acceleration with EMG reached 0.79.c4 The Trier test has people stand and speak, so posture and breath move with the label. The classifier does not need them: physiology without motion or breath carries more than motion does.

The six-item STAI rose by 7.7 points on average from baseline to stress, so people reported feeling the test. Across the 15 people, the rise in reported anxiety did not follow the size of the bodily change: Spearman rho 0.03 with EDA, 0.02 with heart rate and -0.45 with breathing rate (p 0.09).c5 The p values for EDA and heart rate were 0.93 each. With 15 people, a correlation's interval is wide, so this result rules out only a strong relation.

Table 2 sets the forecasts in the first preregistration against the outcomes. The forecast that motion would carry more than physiology, given probability 0.6, was wrong.

The planted result is the general one. It needs no stress physiology: any dataset of overlapping windows with person or block structure will let a random split score the recording rather than the state. The argument between Saeb and colleagues [2] and Little and colleagues [3] was over which split estimates the error that matters. A known-answer test settles a narrower point for this design: when the planted answer is "nothing generalises", the random split says "perfect".

On the real data the inflation was small for the forest (0.033) and larger for logistic regression (0.086). Logistic regression gains the most from scaling each person to their own baseline, which suggests the gap for that model comes from differences in each person's resting level; we did not test that reading separately. The classifier does hold for new people inside this sample, which agrees with the dataset paper's held-out results [1].

The felt-stress result points the same way as the question Campbell and Ehlert [5] put: people reported more anxiety and their bodies changed, but across people the size of the reported rise did not track the size of the bodily change at any strength this sample could detect. A classifier trained on bodily signals reads the laboratory hour; it does not read how much that hour cost a given person.