Participant-Aware Model Validation for Repeated-Measures Data: Comparative Cross-Validation Study

Introduction

Machine learning (ML) is becoming increasingly central to analyzing complex human data gathered through repeated trials, particularly in biomechanics, clinical assessment, and digital health contexts. These datasets, often comprising multiple sessions or sensor recordings per participant, provide insight into within-person variability and strengthen longitudinal evaluations. In biomechanics, repeated gait and hop assessments quantify within-person variability and recovery [1,2]. Similar repeated designs track performance shifts in sports contexts [3].

In clinical and behavioral research, accounting for repeated measures is crucial for making valid inferences. Powell et al [4] applied repeated-measures analysis of covariance to assess how breast support influences running biomechanics while controlling for within-participant variation. Similarly, Keogh et al [5] used repeated-measures correlation to separate individual-level patterns from group-level effects in a cohort of athletes. These examples highlight the necessity of modeling within-participant structures to draw meaningful conclusions.

With the rise of wearable technologies and sensor-based monitoring, repeated-measures data have become increasingly prevalent in applications ranging from cardiac diagnostics to musculoskeletal load tracking [6,7]. When ML evaluation ignores participant identity, trial-level dependencies can leak across folds, inflating performance and undermining generalizability [8-10]. Despite broad awareness of this risk, stratified K-fold CV is still commonly used in ways that overlook subject boundaries. This creates data leakage, as correlated trials from the same participant can appear in both training and test sets. Studies in neuroimaging, ophthalmology, and sensor-based health applications have reported this pattern [11-13]. Our companion standardized rebound side hops (SRSH) study addressed the same clinical construct using the entire biomechanical time series with a 1D convolutional neural network under leave-one-participant-out cross-validation (LOPOCV) [14]. That analysis demonstrated that participant-wise evaluation is crucial for credible accuracy on repeated hop trials. That study held model hyperparameters fixed across folds; here, we systematize the validation question by comparing participant-aware CV designs and, when tuning is required, separating it from evaluation.

Participant-aware validation strategies such as LOPOCV and group K-fold CV have been proposed to enforce participant-level separation during evaluation [15,16]. However, these strategies differ in how they handle intraparticipant variation during hyperparameter tuning, and simple (nonnested) implementations can still introduce selection bias. Nested CV frameworks, which decouple tuning from evaluation, offer a principled alternative to reduce selection bias and overfitting, but adoption remains limited in clinical and behavioral ML [8]. Crucially, the field lacks a systematic benchmark that goes beyond accuracy to quantify (1) validation bias, (2) train-test gaps, (3) participant-level rank stability, and (4) computational efficiency across commonly used models—limitations that constrain reproducibility and practical deployment.

This study addresses this need by providing a systematic, multidimensional benchmark of cross-validation (CV) strategies for repeated-measures clinical ML data. We compare 4 CV designs, including a fully nested framework with LOPOCV for outer-loop evaluation and group K-fold (K=3) for inner-loop model selection, across 10 classifiers and the 4 evaluation axes listed above. The anterior cruciate ligament reconstruction (ACLR) fear-of-reinjury classification task serves as a representative case study; the primary contribution is the validation methodology, which generalizes to other repeated-measures clinical settings. The study follows the TRIPOD+AI (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis—Artificial Intelligence extension) guidelines [17], and the completed checklist is provided in Table S1 in Multimedia Appendix 1. The following research questions were posed to guide the investigation:

• Does standalone LOPOCV overestimate performance by neglecting within-participant variation during tuning?
• Can group K-fold (K=3) CV improve model selection by leveraging repeated trials?
• Does a nested LOPOCV (outer) + group K-fold CV (inner) strategy yield more stable and realistic performance estimates for repeated-measures classification?

Methods

Study Cohort

This study included 72 individuals (43 males and 29 females; age: mean 28.4 years, SD 6.7 years), all between 6 and 24 months after unilateral ACLR using a hamstring autograft.

Ethics Considerations

This study was approved by the Regional Ethical Review Board in Umeå, Sweden (approval numbers 2015/67-31 and 2021-03860). All participants provided written informed consent before participation. Participant data were deidentified before analysis and handled in accordance with applicable Swedish regulations on privacy, confidentiality, and patient safety. Participants did not receive any compensation.

Biomechanical Data Collection

We used the SRSH test to evaluate lateral dynamic stability, neuromuscular control, and lower-limb coordination under sport-specific plyometric loading [18]. During each trial, participants performed single-leg lateral hops across 2 adjacent force plates, with an emphasis on controlled landings and immediate rebounds.

Each participant completed between 5 and 10 valid SRSH trials under standardized protocol conditions. Three-dimensional motion capture was conducted using an 8-camera optical system (Qualisys Oqus 300, 240 Hz), synchronized with dual force platforms (Kistler 9286AA, 1200 Hz). A 56-marker full-body setup captured segmental joint motion.

Data Preprocessing

Marker trajectories were processed in Qualisys Track Manager (version 2019.3) using a zero-lag, fourth-order low-pass Butterworth filter (12 Hz for markers; 50 Hz for force data). The filtered data were exported to Visual3D (C-Motion Inc) for calculation of joint angles and moments via inverse dynamics. All kinetic variables were normalized to body mass.

The stance phase—defined from initial ground contact to the lowest vertical position of the center of mass—was segmented and time-normalized to 101 points using third-order polynomial fitting. Trial-level metadata, including participant identifiers p_i and trial indices t_i, were retained for participant-aware modeling and validation.

Target Labeling: Fear of Reinjury

After preprocessing, fear of reinjury was assessed using item 9 of the 17-item Tampa Scale of Kinesiophobia, which states “I am afraid that I might injure myself accidentally.” Following the classification procedure proposed by Markström et al [19], participants who responded “agree” or “strongly agree” were categorized as having high fear (n=36 participants, 301 trials), and those who responded “disagree” or “strongly disagree” were categorized as having low fear (n=36 participants, 322 trials). These binary categories served as target labels for the classification task.

Feature Engineering and Selection

Time-series descriptors were generated with TSFRESH [20] from 48 biomechanical signals (angles and moments; lateral/medial landings). We retained 13 summary statistics per signal (mean, SD, variance, skewness, kurtosis, 10th/50th/90th percentiles, autocorrelation at lags 1 and 2, counts above/below the mean, and absolute sum of changes), yielding 624 features per trial.

TSFRESH’s built-in hypothesis tests were applied at α=.05 with false-discovery control to remove noninformative descriptors. Surviving features were ranked by Gradient Boosting Classifier (GBC) importance, computed on the pooled training data used for feature preparation. The ranking produced top-k subsets (top 5-10 [step size 1], then 20-100). To keep inputs compact, stable, and fast to evaluate—and to avoid mixing feature selection with performance reporting—we fixed k=10 a priori and used the corresponding top-10 set for all CV experiments. This choice reflects diminishing returns beyond small k on our development trials and keeps the focus on the validation design rather than on feature-count tuning. Importantly, the top-10 feature set was determined before any CV experiment began and was held constant across all folds and strategies. The features were selected using domain-relevant biomechanical reasoning and a single, preexperimental GBC ranking, not through a data-driven search repeated within each fold. As a result, no information from test folds influenced the feature set, eliminating feature-selection bias from the validation comparison. Correlated features (eg, bilateral joint angles) were intentionally retained, as the goal was to evaluate validation-strategy effects under a fixed, realistic feature set rather than to optimize feature independence.

The top-10 ranked features (with signal/statistic provenance) are listed in Table 1 and were held fixed across all models and validation schemes. Within each fold, these features were z-scaled (using StandardScaler from scikit-learn) on the training split and then applied to the corresponding test split. Feature ranks are documented to specify model inputs for the validation comparison. They are not advanced as biomarker evidence; this study aimed to examine participant-aware evaluation and performance inflation when participant identity is ignored. An overview of this feature pipeline and its integration with the evaluation design is shown in Figure 1.

‎

Machine Learning Algorithms and Hyperparameter Tuning

Classifiers

Ten supervised classification models were selected based on their frequent use in biomechanical and clinical ML tasks, representing a range of algorithmic families. These included 3 linear models—logistic regression (LR), linear discriminant analysis (LDA), and quadratic discriminant analysis (QDA); 1 instance-based method—k-nearest neighbors (KNN); and 6 ensemble-based classifiers—random forest (RF), extra trees (ET), AdaBoost (ADA), GBC, Extreme Gradient Boosting (XGBoost), and Light Gradient Boosting Machine (LGBM).

Hyperparameter Handling and Nested Validation

Two hyperparameter strategies were implemented: a fixed configuration using default settings and a tuned configuration using nested CV. In all single-level baselines (stratified 10-fold, group K-fold [K=3], LOPOCV), models used the fixed hyperparameters listed in Table S2 in Multimedia Appendix 1 (right column). Only the nested design performed grid search in the inner loop; no tuning ever used outer-test data. In the fixed configuration, each model was initialized with its default hyperparameters (via a common build_models function), and no grid search was performed. This fixed-parameter setup was used for the 3 single-level CV schemes noted above.

By contrast, the tuned configuration used nested CV, wherein group K-fold (K=3) served as the inner loop for hyperparameter optimization and LOPOCV as the outer loop for evaluation. We chose K=3 to balance 2 constraints. First, each inner training fold required a sufficient number of participants for stable model fitting (approximately 48 of the 71 remaining participants per outer fold). Second, computational cost scales linearly with K in the inner loop and multiplicatively with the 72-fold outer LOPOCV. Hyperparameter tuning was performed using GridSearchCV (scikit-learn) across predefined parameter grids (Table S2 in Multimedia Appendix 1). For each outer fold (held-out participant), the best hyperparameters (denoted θ*) were selected based on inner-loop validation accuracy. A final model was then retrained on the entire outer training set using θ* and evaluated once on the held-out outer participant. This design cleanly decouples tuning from evaluation, minimizing selection bias.

Bias-Variance Handling

In the nested configuration, the inner-loop grid spans the parameters that govern model capacity (bias-variance trade-off). Selection maximizes macro-F₁-score averaged across inner folds; when 2 settings are statistically indistinguishable (Δ≤0.5 pp), ties are broken in favor of the simpler configuration (lower capacity) to reduce variance. We quantify variance at evaluation time via outer-fold dispersion across participants and participant-level rank stability.

CV Methodology

Notation and Definitions

To formalize our dataset and modeling pipeline, we define the following: Let the dataset be denoted as D = {(x_i, y_i, p_i, t_i)}^N_i=1, where x_i is the feature vector; y_i ∈ {0, 1} is the binary class label; represents the predicted class label for trial i; p_i ∈ P is the participant identifier; and t_i is the trial number for the ith observation within participant p_i.

Let P be the set of unique participant IDs with cardinality |P|, and D_p = {(x_i, y_i, t_i)} | p_i = p} be the set of all trials for participant p.

We define M as an ML model instance; θ ∈ Θ as a specific hyperparameter configuration; Θ as the hyperparameter search space; Train(M, D_train) as the training procedure; Eval(M, D_test) as the evaluation procedure (eg, accuracy, F₁-score); and Metric(·) as any function computing a classification metric from predicted and true labels.

Stratified K-Fold CV

In stratified K-fold CV, the full dataset D is randomly divided into k approximately equal folds while preserving the overall class distribution. We use the stratified variant to maintain class prevalence within each fold; this is not a rebalancing method but a splitting rule that preserves label proportions while remaining trial-wise. For each fold j ∈ {1, ..., k}, the data are partitioned into a training set D^(j)_train and a test set D^(j)_test. The model is trained on D^(j)_train, evaluated on D^(j)_test, and the process is repeated for all folds. Final performance is computed as the average across all folds.

We used StratifiedKFold (k=10, shuffle=True, random_state=42) from scikit-learn. While commonly used, this method does not control for participant identity and may introduce data leakage in repeated-measures datasets (see Figure 2A). The full pseudocode is provided in Algorithm S1 in Multimedia Appendix 1.

‎

Leave-One-Participant-Out Cross-Validation

LOPOCV evaluates true interparticipant generalization by holding out all trials from a single participant in each fold. For each participant p ∈ P, the model is trained on the dataset D^(p)_train = D\D_p, where D_p contains all trials from participant p. The trained model is then evaluated on D_p, ensuring no overlap between training and testing data.

We implemented LOPOCV using LeaveOneGroupOut from scikit-learn, treating participant ID as the grouping variable. This method enforces participant-level separation but does not address selection bias during hyperparameter tuning (see Figure 2B). An explicit step-by-step description is provided in Algorithm S2 in Multimedia Appendix 1, including participant-averaged metrics and train-test gap calculation.

Group K-Fold CV

Group K-fold (K=3) CV preserves participant-level independence by splitting the set of participants P into k disjoint subsets P₁, ..., P_k. For each fold j, all data from participants in the group P_j form the test set D^(j)_test, while the remaining data constitute the training set D^(j)_train. The model is trained and evaluated accordingly.

We used GroupKFold (n_splits=3) to ensure that all trials from each participant remained in the same fold. This method maintains participant-level separation during training and testing while allowing multiple participants per fold. Compared with LOPOCV, group K-fold (K=3) offers greater computational efficiency but does not fully isolate tuning from evaluation (see Figure 2C). Implementation details are provided in Algorithm S3 in Multimedia Appendix 1.

Combined Approach: Nested LOPOCV (Outer) With Group K-Fold CV (Inner)

To support robust hyperparameter tuning without introducing selection bias, we implemented a nested CV framework. The outer loop used LOPOCV, while the inner loop used group K-fold (K=3) CV for tuning.

In the outer loop, for each participant p, the test set D^(p)_{outer_test} comprised all trials from that participant. The remaining data D^(p)_{outer_train}=D\D_p was used for training and tuning. Within this training set, participants P\{p} were split into 3 groups for the inner loop.

The inner loop performed a 3-fold CV across these remaining participants. For each inner fold j, the training and validation sets were defined as follows:

• Inner validation set:D_val^(p,j) = ∪_p′∈p(j)D_p′
• Inner training set:D_train^(p,j) = D_{outer_train}^(p)\D_val^(p,j)

Each model M_θ^(p,j) was trained with a candidate hyperparameter configuration θ ∈ Θ and evaluated on D_val^(p,j). The optimal configuration θ_p* was selected based on mean inner validation performance:

Using θ_p*, the final model M^(p) was trained on D_{outer_train}^(p) and evaluated on D_{outer_test}^(p).

This nested design ensures that hyperparameter tuning is fully decoupled from performance estimation by isolating inner tuning folds from outer evaluation folds. Figure 2D illustrates the complete nested validation structure. The nested, participant-aware framework is specified in Algorithm S4 in Multimedia Appendix 1.

The hyperparameter grids used in this process were deliberately designed to be capacity-aware, enabling the inner CV to explicitly explore low-bias/high-variance versus high-bias/low-variance regimes (see Tables S2 and S3 in Multimedia Appendix 1 for the capacity-controlling parameters).

Performance Evaluation Metrics

We assessed model performance using standard classification metrics derived from the confusion matrix, based on counts of true positives (TPs), true negatives (TNs), false positives (FPs), and false negatives (FNs). The following metrics were computed for each outer test fold in every CV scheme:

Accuracy = (TP + TN)/(TP + TN + FP + FN)

Precision = (TP)/(TP + FP)

Recall = (TP)/(TP + FN)

F₁-score = 2 × (precision × recall)/(precision + recall)

We calculated these metrics from the test-fold predictions of the outer loops for all 4 CV schemes: stratified K-fold, LOPOCV, group K-fold (K=3), and nested LOPOCV.

Participant Aware Summary Metrics

Summary Metrics for Participant-Aware Validation Strategies

For participant-aware strategies (LOPOCV and nested LOPOCV), we computed 2 complementary summaries:

Mean LOPOCV Performance (Participant-Averaged)

This metric averages model performance across all participants, giving equal weight to each individual, regardless of the number of trials:

where P is the total number of participants; and D^(p)_test denotes the outer test set for participant p.

Overall Performance (Aggregated Level)

This summary pools all test predictions and true labels across outer folds and then computes each performance metric on the combined dataset:

Overall metric = Metric(∪_p=1^P{i ∈ D_test^(p)})

The mean LOPOCV performance emphasizes cross-participant stability, whereas overall performance reflects population-level predictive accuracy by weighting participants according to trial count.

Train-Test Gap: Generalization Stability

To assess generalization stability and potential overfitting, we computed the train-test gap for each outer fold in the LOPOCV and nested CV frameworks. This gap quantifies the difference in model accuracy between the outer training and test sets.

Let θ* ∈ Θ denote the selected hyperparameter configuration—either default (in the fixed configuration) or optimized via inner-loop group K-fold tuning (in nested CV). Let M_θ* be the trained model using this configuration. For each outer fold f, corresponding to the held-out participant p, the train-test gap is defined as follows:

Gap^(f) = Accuracy[M_θ*, D^(f)_train] – Accuracy[M_θ*, D^(f)_test]

This metric quantifies how well a model’s training performance carries over to unseen test participants. Large gaps indicate potential overfitting, whereas small gaps suggest stable generalization.

To summarize overall generalization behavior, we report the mean and SD of the train-test gap across all participants for each model m:

This metric reflects how well training performance translates to unseen test participants. We did not compute the train-test gap for stratified K-fold or group K-fold CV, as both methods allow trials from the same participant to appear in both the training and test sets, thereby violating independence and compromising the interpretability of generalization behavior.

Bias Estimation Across Validation Strategies

To quantify how CV strategies deviate from participant-aware evaluation, we compared strategy-level summary metrics with a participant-aware baseline. The baseline reference was either LOPOCV or nested LOPOCV, as indicated in each comparison. For each metric m ∈ {accuracy, F₁-score, precision, recall, Matthews correlation coefficient}, bias was computed as follows:

where represents the participant-averaged or overall score (as specified) obtained using stratified 10-fold or group K-fold (K=3) CV; and represents the corresponding LOPOCV- or nested LOPOCV–based estimate.

A positive bias implies that the strategy overestimates model performance relative to the participant-aware baseline, whereas a negative bias indicates underestimation. To quantify overall deviation across metrics, we also computed the mean absolute bias (MAB) for each model:

MAB reflects the average magnitude of deviation from the baseline, providing a scalar summary of validation reliability. Larger MAB values indicate greater deviation from a realistic participant-aware estimate and, thus, lower external validity. Lower MAB values indicate that the CV strategy yields results more consistent with robust, participant-separated evaluation.

This procedure is consistent with earlier findings that participant-independent validation frameworks, such as LOPOCV, produce more accurate out-of-sample performance estimates, particularly in domains involving interindividual variability [15,21,22].

Model Ranking Method

Overview

To evaluate the relative performance of classifiers and assess ranking stability across individuals, we implemented a nonparametric model-ranking procedure, following established practices in classifier evaluation [23,24].

Let A_p,m denote the classification accuracy of the model m for participant p, computed using either LOPOCV or nested LOPOCV. The ranking procedure involved 2 steps.

Step 1: Within-Participant Ranking

For each participant p, all models were ranked based on their accuracy A_p,m, from highest (rank=1) to lowest (rank=10). Ties were resolved by assigning the average of the spanned ranks.

Step 2: Across-Participant Summary Statistics

For each model m, we computed the following summary metrics over all participants:

Lower values indicate better average performance across participants.

The aforesaid equation reflects consistency of ranking across participants.

Models with lower mean ranks and smaller rank variability were considered more robust and consistent across individuals. Ranking analyses were conducted independently for LOPOCV and nested CV frameworks to evaluate how hyperparameter tuning influences ranking stability.

Computational Efficiency Estimation

Computational Efficiency Metrics for Model Deployment Feasibility

To assess the feasibility of deploying each model configuration, we evaluated computational efficiency using 3 metrics recorded during outer-loop validation:

Training Time Per Outer Fold (Seconds)

Training time per outer fold is defined as the elapsed wall-clock time required to train each model on the outer training set. For nested LOPOCV, this measure includes the full runtime of the inner-loop hyperparameter optimization using GridSearchCV. For LOPOCV, group K-fold (K=3) CV, and 10-fold CV, only the time required for model training (without tuning) was recorded.

Inference Time Per Sample (Milliseconds)

Inference time per sample is computed as the average time required to generate a single prediction. This value was averaged over all test samples across outer folds. It indicates the suitability of models for real-time or near–real-time applications.

Model Size (Megabytes)

Model size is estimated as the in-memory serialized size of each trained model instance using pickle.dumps(). This metric reflects storage requirements and transferability for deployment in clinical or embedded systems.

All efficiency metrics were automatically logged during model evaluation for each model-CV combination. Comparisons across configurations enable assessment of trade-offs among performance, complexity, and ease of deployment.

Computational Environment and Reproducibility

All analyses were performed using Python 3.10.12 (Python Foundation) on a high-performance workstation equipped with an Intel Core i9-13900K CPU, an NVIDIA GeForce RTX 4090 GPU, and 64 GB of DDR5 RAM. The software environment included scikit-learn version 1.2.2 [25], XGBoost version 1.4.2 [26], LGBM version 4.3.0 [27], and MLflow 2.10.2 for experiment tracking and reproducibility.

All training runs, CV folds, hyperparameter configurations, and resulting artifacts were managed and logged using MLflow, which provided full experiment lineage and enabled structured comparison across model-validation combinations. Each run recorded parameter settings, metric scores, training durations, and serialized model outputs, enabling seamless integration with the reported analysis.

Randomized operations, including data shuffling, CV splitting, and model initialization, were controlled using a fixed seed (random_state=42). All splitters and estimators were instantiated with this seed where applicable; CV groups were specified as participant IDs to enforce participant integrity.

Results

Model Performance Across Validation Strategies

Table 2 presents overall and participant-averaged accuracy for each classifier across 4 validation strategies. Performance varied by model type and validation scheme.

The highest accuracy values for the ensemble models ET, GBC, RF, and XGBoost were obtained under 10-fold CV. For example, ET achieved 0.91 accuracy under 10-fold CV and 0.66 under LOPOCV. LR and LDA exhibited relatively stable accuracy across validation methods, with differences ≤0.03 between LOPOCV and 10-fold CV.

Accuracy estimates under nested CV closely matched those from LOPOCV, differing by ≤0.02 across all classifiers. Group K-fold (K=3) CV produced intermediate accuracy scores, typically falling between LOPOCV and 10-fold CV. Full evaluation metrics (accuracy, F₁-score, precision, recall, and Matthews correlation coefficient) for each model are provided in Table S5 in Multimedia Appendix 1.

Train-Test Gap and Overfitting Risk

Figure 3A summarizes the mean (SD) train-test gaps for each classifier under LOPOCV and nested LOPOCV; exact values are provided in Table S4 in Multimedia Appendix 1. Gaps ranged from 0.08 to 0.36 across models. Ensemble methods (ET, RF, and GBC) showed the largest gaps (≥0.34) under both validation strategies, whereas LR and LDA were approximately 0.08. Gap estimates were similar for LOPOCV and the nested scheme, with no consistent differences across models. Figure 3B shows participant-level distributions. Ensemble models displayed wider IQRs and more outliers, whereas linear models were more tightly clustered around their medians, consistent with Figure 3A.

‎

‎

Bias Between Validation Strategies

Bias (Δ) for all classifiers is reported in Table S6 in Multimedia Appendix 1 and visualized in Figure S1 in Multimedia Appendix 1 (diverging scale centered at 0). Across classifiers, 10-fold CV exhibits the largest positive Δ relative to participant-aware baselines, consistent with performance inflation when trials from the same participant appear in both training and test sets. By contrast, LOPOCV, nested LOPOCV, and group K-fold yield Δ values approximately 0 across metrics. For example, KNN shows marked increases relative to LOPOCV under 10-fold CV (accuracy +0.22, Matthews correlation coefficient +0.82, MAB 0.36 in Table S6 in Multimedia Appendix 1), whereas LR shows no measurable difference between nested LOPOCV and Group K-fold.

Model Ranking Consistency Across Participants

Table 3 summarizes participant-level rank statistics for each classifier under LOPOCV and nested CV. For each model, the table displays the mean rank, SD, best rank, and worst rank across all participants. Lower rank values correspond to higher participant-level accuracy. Across both validation strategies, ET yielded the lowest mean ranks (5.19 for LOPOCV and 5.39 for nested CV), whereas LGBM had the highest (6.05 and 6.06, respectively). SDs ranged from 1.89 (RF, LOPOCV) to 2.79 (ADA, nested CV).

All classifiers achieved a best rank of 1.0 for at least one participant, with fractional values (eg, 1.5) indicating ties averaged according to standard conventions. The worst rank was uniformly 10.0 for all classifiers. Differences in rank between LOPOCV and nested CV were small. For 8 of the 10 models, the absolute difference in mean rank was ≤0.25. Changes in rank variability (SD) were also limited, with maximum observed shifts of 0.31 in mean rank (XGBoost) and 0.22 in SD (ADA).

Computational Efficiency Analysis

Table 4 presents the average training time per outer fold, inference time per sample, and final model size for each classifier under 4 validation strategies. Training time varied by more than 4 orders of magnitude across models. The fastest models—KNN, QDA, LDA, and LR—completed training in under 0.02 seconds per fold across all validation strategies. LGBM required 275 seconds per fold under nested CV and 125 seconds under 10-fold CV, even without tuning. XGBoost was more efficient than the other 3 ensembles during tuning, requiring approximately 3.7 seconds per fold.

Inference times were uniformly low. All models classified a test sample in ≤2 milliseconds per sample. ET and RF had nonzero inference times (2 ms), but all models were well within real-time application thresholds. Model sizes remained modest. LR had the smallest footprint (~1 kB). ET under nested CV had the largest size (~4 MB), followed by RF and GBC. LGBM maintained a small model size (~0.23 MB) despite its long training time.

Discussion

Principal Findings

The main finding of this study is that CV strategy substantially influences reported model performance in repeated-measures classification. Stratified 10-fold CV systematically overestimated accuracy (eg, ET: 0.91 vs 0.66 under LOPOCV) due to participant-level data leakage. Participant-aware strategies (LOPOCV, group K-fold, and nested CV) produced more conservative and stable estimates. The nested LOPOCV + group K-fold framework achieved the best balance among unbiased evaluation, participant-aware separation, and hyperparameter tuning decoupling, with minimal additional bias compared with standalone LOPOCV.

Theoretical Rationale for Participant-Aware Validation

Accurate model evaluation in repeated-measures datasets requires validation strategies that respect the nested data structure, specifically the dependency among trials within each participant. Classical CV methods assume independent and identically distributed observations, an assumption that is violated when multiple measurements come from the same individual.

Stratified K-fold CV maintains class balance across folds but splits individual trials randomly, often placing multiple trials from the same participant in both the training and test sets. This introduces participant-specific information leakage, which in turn inflates reported performance metrics and misrepresents generalizability [8,10,15].

LOPOCV addresses this leakage by holding out all trials from a single participant in each fold. This ensures participant-level independence during evaluation [8]. However, LOPOCV does not prevent selection bias: if hyperparameter tuning uses data from the same participants later used for evaluation, model selection remains compromised. Group K-fold (K=3) CV offers a partial solution by assigning all trials from a participant to the same fold; however, when used for both tuning and evaluation, it still allows indirect bias to persist.

To resolve both leakage and selection bias, this study implemented a nested CV structure: LOPOCV for outer-loop evaluation and group K-fold (K=3) for inner-loop tuning. This structure ensures that evaluation is based on participants unseen during training and tuning. Furthermore, tuning leverages repeated trials from different individuals, handling intrasubject variability without reintroducing test data. This approach aligns with established recommendations for unbiased evaluation in supervised learning [15,16,28].

A key rationale behind this design lies in recognizing that, in repeated-measures contexts, the effective sample size approximates the number of participants, not the total number of trials [29,30]. Trial-level CV methods that ignore this dependency systematically underestimate generalization error and lead to inflated performance metrics, particularly for models that exploit individual-specific patterns.

Effects of Validation Strategy on Model Performance

Stratified 10-fold CV produced the highest reported accuracies but inflated generalization estimates, with differences of up to 0.25 compared with LOPOCV (eg, ET: 0.91 vs 0.66). Among participant-aware methods, high-capacity classifiers (KNN, ET, and GBC) still exhibited train-test gaps ≥0.30 under standalone LOPOCV, suggesting overfitting to participant-idiosyncratic patterns when tuning was not decoupled from evaluation.

Group K-fold (K=3) preserved participant identity while reducing computational burden, with performance comparable to LOPOCV and stable rankings. Nested CV produced the most cautious and consistent estimates across classifiers: it minimized validation bias, narrowed train-test gaps, and maintained ranking consistency, yielding a representative estimate of out-of-sample performance. A related SRSH study using participant-wise evaluation reported 75.6% accuracy, which aligns with the participant-aware range observed here and contrasts with the optimistic trial-wise figures [14].

These findings answer research questions 1 and 2:

• Research question 1: LOPOCV reduces leakage but may not prevent overfitting when intrasubject variability is not managed.
• Research question 2: A nested participant-aware framework provides the most robust and unbiased evaluation of model generalization in repeated-measures data.

Model Stability and Ranking Consistency

Although the top classifiers showed comparable average accuracies, their rankings differed considerably at the individual participant level. Specifically, ET and RF demonstrated greater ranking stability across participants compared with LGBM and ADA, which showed higher variability. Despite these individual differences, rankings remained highly stable between LOPOCV and nested CV. For 8 out of 10 models, the absolute difference in mean rank between the 2 schemes was ≤0.25, and the change in rank SD was ≤0.17, supporting the conclusion that participant-aware tuning does not distort model comparisons.

These findings directly address research question 3, emphasizing the practical value of incorporating participant-level validation frameworks when deploying ML models in real-world scenarios. While model stability is essential, computational demands must also be considered when selecting validation strategies for deployment.

Efficiency and Practical Considerations

Validation strategies varied considerably in computational cost. Nested CV incurred the highest training time, especially for ensemble models (eg, LGBM: >275 seconds/fold), due to inner-loop hyperparameter tuning. By contrast, linear models (eg, LR and LDA) trained in under 0.02 seconds per fold, underscoring the substantial overhead introduced by nested designs. Group K-fold (K=3) CV offered a computationally efficient compromise, producing performance estimates comparable to LOPOCV with minimal bias while avoiding the full tuning cost of nested CV.

Despite differences in training time, all models exhibited low inference latency (≤2 ms/sample) and small memory footprints, making them suitable for deployment in real-time settings once trained. In practice, training efficiency—not inference cost—should guide validation strategy selection, particularly in clinical or iterative modeling contexts where scalability and runtime constraints are critical.

Limitations and Future Work

This study was designed to isolate the impact of validation strategy on performance estimation rather than to maximize predictive accuracy. Several limitations should be acknowledged. First, the analysis was restricted to manually engineered features and narrow hyperparameter grids. While this ensured consistency across models, broader, systematically documented optimization techniques, such as Bayesian search or automated feature extraction, may yield different performance rankings. Second, the dataset was limited to a binary classification task in 72 individuals (43 males and 29 females; mean age 28.4 years, SD 6.7 years) following ACLR with hamstring autograft, all recruited 6-24 months after surgery. The extent to which the observed validation discrepancies generalize to cohorts with different demographic profiles (eg, slightly older populations, different graft types, time since surgery, or varying preinjury activity levels) remains an open question. Future studies should also test generalizability in multiclass problems, longitudinal predictions, and additional clinical contexts. Third, although the study included a range of linear and ensemble learning algorithms, it did not evaluate deep neural networks, which may be more sensitive to overfitting and leakage in repeated-measures data. Applying participant-aware nested validation to recurrent or convolutional neural networks, particularly in high-dimensional time-series biomechanics, remains a critical direction for future research. Fourth, this study did not address probability calibration or algorithmic fairness, both of which are essential for the deployment of clinical ML. Future work should integrate calibration metrics (eg, Brier score, calibration curves) and evaluate fairness-aware performance under participant-aware validation schemes. Finally, feature-level interpretation was out of scope. We focused on evaluating models rather than identifying which features drive predictions. Importance estimates depend on the validation design and may be unstable under trial-wise splits. We therefore reserve feature attribution for future work conducted strictly under the nested, participant-aware protocol established here, ensuring that any reported importances reflect leakage-free evaluation.

Comparison With Previous Literature

The limitations of stratified CV in repeated-measures settings have been well documented. Early theoretical work identified selection bias and performance inflation when training and evaluation are not properly separated [9,15]. These studies laid the foundation for participant-aware evaluation, though they did not explicitly address repeated measures or participant-specific dependency structures.

More recent applied studies in neuroimaging, biomechanics, and mobile health have adopted LOPOCV or group CV to control for participant-level leakage (eg, [8,9]). However, many of these implementations lack full tuning-evaluation decoupling, often using the same CV scheme for both hyperparameter selection and final testing. This results in partial leakage and residual bias, even when participants are held out during the final evaluation.

Prior work has noted the risk of data leakage from trial-wise CV and the value of participant-aware evaluation [8,9,15]. However, no study has systematically benchmarked a fully nested CV design (LOPOCV outer loop or group K-fold inner loop) for repeated-measures classification in biomechanics across multiple classifiers and evaluation dimensions. This study provides such a benchmark, quantifying classification performance, validation bias, train-test generalization, participant-level ranking stability, and computational efficiency across 4 strategies and 10 classifiers. A comparative overview of previous methodologies and their limitations is provided in Table 5, highlighting how this study addresses critical gaps in the current literature.

Regulatory Relevance

Subject-aware validation aligns with emerging regulatory expectations for clinical ML systems, although direct regulatory conclusions should not be drawn from a single case study. The nested CV framework demonstrated here ensures subject-aware separation in both evaluation and hyperparameter tuning, reducing the risk of performance inflation, an issue highlighted in regulatory and ethical reviews of artificial intelligence (AI) systems. Several regulatory documents and frameworks explicitly support these principles:

• TRIPOD+AI [17] calls for complete and reproducible reporting of validation procedures that mimic intended use scenarios.
• The FDA’s Good Machine Learning Practice guidelines [43] stress the need for strict separation between training and test data, comprehensive documentation of data lineage, and bias minimization.
• The EU Medical Device Regulation (2017/745) [44] and the proposed Artificial Intelligence Act (2023) [45] categorize clinical AI tools as high-risk technologies, requiring developers to present robust performance evidence and demonstrate algorithmic transparency.

By aligning with these standards, the nested CV approach not only enhances internal validity but also contributes to regulatory readiness. It provides a principled validation pathway aligned with international expectations for safety, accountability, and trustworthiness in AI-driven medical tools.

Summary of Implications

This study demonstrates that the validation strategy critically influences model trustworthiness in repeated-measures classification. Trial-level CV overstates performance by ignoring participant dependencies, while subject-aware methods, particularly nested CV, offer more realistic and unbiased estimates of generalization. When full nesting is computationally infeasible, group K-fold (K=3) CV offers a practical alternative, maintaining participant-level separation with minimal validation bias. In deployment scenarios, where model outputs influence clinical or behavioral decisions, variability in participant-level predictions may undermine reliability.

While this study used balanced class distributions, real-world applications often involve class imbalance and prevalence shifts. In such contexts, metrics such as positive predictive value and negative predictive value may vary despite stable sensitivity and specificity [46]. Future implementations should consider calibration and context-specific performance metrics to ensure reliability under deployment conditions.

Conclusions

This study highlights the critical role of validation strategy in evaluating ML models trained on repeated-measures data. Through a comparative evaluation of 4 CV methods—stratified 10-fold CV, group K-fold (K=3) CV, LOPOCV, and nested LOPOCV—we demonstrate that standard approaches often overestimate model performance by failing to account for participant-level dependencies.

Nested, subject-aware validation provided the most reliable generalization estimates in this repeated-measures setting, and our efficiency analysis indicates viable lower-cost options when full nesting is impractical. Classifier stability, train-test gap, and computational efficiency were all found to vary substantially across validation schemes, showing that validation design materially affects performance estimates relevant for deployment.

These results have practical implications for clinical model development, where decisions based on inflated metrics may compromise safety and reproducibility. The nested subject-aware framework proposed here offers a principled path toward transparent and trustworthy ML systems aligned with current regulatory expectations in health care and other domains characterized by repeated-measures designs. Subject-aware validation is not merely a methodological refinement but a practical necessity for developing reproducible, generalizable, and deployment-ready ML systems in biomechanics, behavioral research, and clinical monitoring.