Machine-Learning Predictive Tool for the Individualized Prediction of Outcomes of Hematopoietic Cell Transplantation for Sickle Cell Disease: Registry-Based Study

Introduction

The complications of sickle cell disease (SCD) can be prevented or ameliorated by disease-modifying therapies [1], but hematopoietic cell transplantation (HCT), and more recently, gene therapy remain the only therapeutic options with curative intent [2-12]. Population-level studies demonstrate the association of outcomes of HCT with age, type of donor, type of conditioning, and graft-versus-host disease (GVHD) prophylaxis [5,6], but do not address the uncertainty regarding individualized outcomes of HCT. Such uncertainty contributes to the decisional dilemma and is a barrier to shared decision-making. An individualized prediction model that incorporates all predictive variables and provides individualized estimates of key outcomes of HCT of interest to patients and their physicians has the potential to inform shared decision-making [13-15]. Brazauskas et al [16] have proposed a predictive model based on the age of the recipient and the type of donor. However, their model does not incorporate other clinically relevant patient, HCT, and disease characteristics and does not include all key outcomes. Computational machine learning (ML) has the potential to determine generalizable predictive patterns and quantify uncertainty, but published ML predictive models for HCT are limited to predicting single clinical outcomes [17-30]. To address the knowledge gap, we developed and described the initial validation of SPRIGHT (Sickle Cell Predicting Outcomes of Hematopoietic Cell Transplantation), an individualized ML prediction model for outcomes of HCT for SCD, incorporating multiple relevant features to make predictions of key clinical outcomes.

Methods

Dataset

We developed SPRIGHT using an anonymized HCT for the SCD dataset [31] derived from data submitted to the Center for International Bone Marrow Transplant Research (CIBMTR) registry on children and adults undergoing HCT for SCD between 1991 and 2021 in the United States. The dataset was obtained through the NHLBI (National Heart Lung and Blood Institute) Biologic Specimen and Data Repository Information Coordinating Center (BIOLINCC) [32].

The CIBMTR maintains a research database to serve as a comprehensive data source that can be used to study cellular therapies, including HCT. All US transplant centers are required to submit outcomes data on all allogeneic transplants when either the stem cell donation or the transplant occurs within the United States. CIBMTR assigns patients to either a Transplant Essential Data (TED) track, which collects core transplant data, or a Comprehensive Report Form (CRF) track that captures detailed disease- and treatment-related data [31]. Assignment to each track is based on submission of the initial pretransplantation TED 2000032419 form and uses a weighted randomization algorithm designed to produce a cohort representative of current clinical practice. All centers submit a Pre-HCT TED Form (Form 2400) for each allogeneic (related or unrelated) HCT.

Of 1641 patients undergoing HCT for SCD between 1991 and 2021, on whom data were submitted to CIBMTR, detailed CRFs were submitted on 763 patients. Of the patients in the dataset, 84% (1377/1641) had undergone HCT after 2007. We performed the imputation of missing data using MissForest, an ML data imputation algorithm that operates on random forest (RF).

Feature Selection

We identified overall survival (OS), event-free survival (EFS), graft failure (GF), acute graft-versus-host disease (AGVHD), and chronic graft-versus-host disease (CGVHD) as key outcomes. We used wrapper methods of backward feature elimination (BFE) and forward sequential selection (FSS) to select and optimize the input variables [33,34]. The BFE procedure begins with a complete set of features and a chosen ML model. The model is trained, and the importance of each feature is evaluated based on the model’s coefficients or feature importance scores, and then the least important feature is discarded, and the model is retrained on the remaining features. This process is repeated until a predetermined number of features is reached or until further removal of features leads to a significant decrease in model performance. Using FSS, we incrementally built a feature set starting with an empty model, sequentially adding a feature that most improves the model performance at each step, as evaluated through a predefined metric like cross-validation score. We continued this stepwise addition until new features no longer significantly enhanced the model or a specified number of features was reached. We have included the detailed descriptive statistics and missingness for each of the selected features and outcomes of interest in Table S3 in Multimedia Appendix 1 and Table S4 in Multimedia Appendix 2

Model Design

The model was designed by subsampling the majority class to create subtraining datasets, followed by pooling and thresholding to obtain the final prediction [35]. We determined the discriminative performance of the model, which refers to how well the predictions can separate between 2 groups of participants, that is, those with or without an outcome. Discrimination was quantified by the concordance (c) statistic (index), which for binary outcomes, is equivalent to the area under the curve (AUC). We also assessed accuracy, that is, the percentage of correct predictions out of all predictions correct or incorrect in the model. We also assessed balanced accuracy, which is an accuracy adjusted for imbalance and is derived by averaging sensitivity and specificity, so that each class’s importance is equal. We compared RF, extreme gradient boosting, logistic regression, Naive Bayes, AdaBoost, and support vector classification algorithms (Table 1). The HCT for SCD dataset spans children and adults undergoing HCT between 1991 and 2021. During this time, there have been many changes in conditioning regimens and improvements in supportive care. Gluckman et al [6] reported that EFS was higher in patients who underwent HCT in or after 2007 as compared to those who underwent HCT in or before 2006 (HR [hazard ratio] 0.95, CI 0.90‐0.99; P=.01). To determine if the model performed consistently across eras, we tested model performance in patients with the year of HCT <2007 versus HCT >2007. We also tested model performance across age at HCT <10, <18, and >18 years, respectively, including outcomes at 1- and 3-year post-HCT (Table 2).

The accuracy of risk estimates, relating to the agreement between the estimated and observed number of events, is called “calibration [36,37].” Calibration is crucial in predictive algorithms because it ensures the accuracy of risk estimates, which directly affects clinical decision-making and patient expectations. Poor calibration can lead to systematic overestimation or underestimation of risk, resulting in false expectations and potentially harmful decisions [36]. We performed causal isotonic calibration, a novel nonparametric method for calibrating predictors of heterogeneous treatment effect [37]. We performed a 5-fold internal cross-validation on the training set to determine the optimal calibration. The resultant calibration model was then applied to the predictions during evaluation. To adjust for the bias caused due to undersampling, we recalibrate the probabilities according to the method by Pozollo et al [38]. We evaluated the calibration curve, slope, and intercept across different outcomes of interest in the post-2007 data (Table 3).

To understand the contributions of each feature to the predictive model, we use the Shapley additive explanations (SHAP) scores (Multimedia Appendices 3-7). SHAP scores are based on game theory’s Shapley values, quantifying each feature’s marginal contribution to individual predictions. They are calculated by measuring how each feature affects the model output when included or excluded from all possible feature combinations.

The Problem of Imbalance

The outcomes data for HCT for SCD is imbalanced, with very few negative outcomes. This imbalance has the potential to lead to a prediction bias, where an uncorrected model default may be skewed toward predicting positive outcomes. To address the problem of imbalance, we constructed a training dataset taking each outcome as variable of interest. We included randomly sampled positive outcomes, typically 1.5‐3 times the total instances of the variable of interest. To address the issue of class imbalance, we used a 2-step approach involving bootstrapping and consensus-based decision-making. Initially, we generated 20 bootstrapped datasets by undersampling the majority class to achieve a (2-3):1 ratio with the minority class. These datasets served as the training sets for our predictive models, ensuring a balanced representation of classes during model training. Once the models were trained, each was then tested on a consistent test dataset to obtain a series of predictions. These individual predictions were subsequently pooled across all models. A final prediction model for each test instance was determined based on a consensus threshold. If the majority of the models exceeded a predetermined threshold agreed on a particular class, that class was assigned as the outcome for the instance. We ran the test dataset and used a RF algorithm on a 2-times repeated 10-fold cross-validation to demonstrate our model’s versatility and response to unknown data (Figure 1). We assigned value 1 for a negative outcome prediction and −1 for a positive outcome prediction and found the average sum across the 20 trials for each element.

Throughout the paper, we are guided by the CREMLS (Consolidated Reporting of Machine Learning Studies) guidelines [39], to ensure transparency and rigor in reporting. We have attached a completed author CREMLS checklist.

‎

Ethical Considerations

The Institutional Review Board at Yale University determined on March 3, 2033, that this study did not constitute human participants research per IRB protocol number 2000032419.

Results

Feature Selection and Optimization

Of the 160 variables in the dataset, we selected 31 potentially clinically relevant predictive variables. Through FSS and BFE processes, we selected a final set of 17 predictive variables grouped into 3 categories. Patient Data variables included age at transplant, age group at transplant, sex, Karnofsky or Lansky score, HCT-comorbidity index, and recipient CMV serostatus. SCD variables included Number of ACS syndromes within 2 years pre-HCT, Required exchange transfusion, Vaso-occlusive crisis needing hospitalization in 2 years pre-HCT, and Hospitalization frequency for vaso-occlusive crises. Transplant Data variables included Donor type, Graft type, Conditioning intensity, Conditioning regimen, Serotherapy (ATG or Alemtuzumab), GVHD prophylaxis, and Donor-recipient HLA matching.

Evaluation Outcomes and Model Performance

Discrimination in predictive performance is evaluated using accuracy, the percentage of correct predictions out of all predictions, balanced accuracy, the average sensitivity and specificity, and each class’s importance is equal, and AUC, the measure of a model’s true positive rate against a false positive rate, indicates the ability to differentiate classes. AUC is the metric used in published literature. The RF model achieved the highest predictive AUC (Table 1) across multiple clinical outcomes. We measure our model’s performance using the benchmark established in the literature. To determine the statistical validity, we implemented the method proposed by Bouckaert and Frank [40]. We first performed 2×10 repeated cross-validation and obtained the performance of each of the models for each fold. We then applied Nadeau and Bengio’s [41] correction, which accounts for training set overlap in the variance estimation, to check whether the mean AUC of the RF is greater than the mean AUC of the other models across folds. The differences in the mean AUC were statistically significant for EFS, OS, GF, AGVHD, and CGVHD (P<.05). We performed hyperparameter tuning using grid search cross-validation for the RF model. The ideal hyperparameters are described in Table 4. AUC, accuracy, and balanced accuracy equaled or exceeded the benchmarks of the ML predictive tools in the published literature [16,42]. The RF model has been previously reported to have the best AUC in predicting survival following HCT [24]. Brazauskas et al [16] published benchmark AUC for EFS of 0.72, and Taheriyan et al [42] reported benchmarks AUC 0.82 for AGVHD post-HCT. Accuracy and balanced accuracy were excellent for EFS (0.76, 0.69), OS (0.82, 0.68), GF (0.8, 0.71), vaso-occlusive pain post-HCT (0.9, 0.78), stroke post-HCT (0.92, 0.65), acute GVHD (0.71, 0.60), CGVHD (0.72, 0.63).

SPRIGHT retained high AUC in subpopulations, including patients ≤10, ≤18, >18 of age in undergoing HCT after 2007, as well as 1- and 3-year survival analysis (Table 1).

Calibration is the agreement between the estimated and observed number of events, for major outcomes. A calibration slope of 1 and an intercept close to zero are associated with good calibration. SPRIGHT revealed good calibration with a slope range of 0.87‐0.96, with small negative intercepts (−0.01 to 0.03), for 4 out of the 5 outcomes. However, OS exhibits nonideal calibration and may be reflective of the overall high OS in all subgroups (Figure 2).

Feature importance analysis using SHAP values revealed consistent patterns across all outcomes. Age at transplantation and donor type emerged as the most influential predictors, corroborating previous findings by Brazauskas et al [16] and Eapen et al [5]. Disease severity indicators, most importantly the frequency of acute chest syndrome episodes in the 2 years preceding HCT, were identified as another critical predictor. The frequency of VOC requiring hospitalization and the need for exchange transfusions also demonstrated substantial predictive importance. This suggests the utility of including pretransplant disease characteristics for predicting outcomes. The comprehensive SHAP analysis, including feature importance rankings and their relative contributions to model predictions, is presented in Multimedia Appendices 3-7.

To demonstrate the clinical utility of our model, we analyzed predictions across 3 distinct hypothetical patient scenarios. Case-specific patient characteristics and their corresponding predicted outcomes are detailed in Table 5 Table S3 in Multimedia Appendix 8 respectively. The model’s predictions aligned with established clinical observations, showing less favorable outcomes in cases involving non-HLA identical donors and in older patients with more severe disease characteristics, which were consistent with previous studies [5,16].

‎

SPRIGHT User Interface

Age is the only numeric feature that is entered manually. The rest of the inputs are categorical and are entered by selecting an option from a drop-down menu. Patient-specific, disease-specific data, and treatment donor, one of the major published treatment regimens or a customized conditioning regimen, conditioning intensity, and GVHD prophylaxis ATG/Alemtuzumab data can be entered. The tabular output describes predicted OS, EFS, GF, Death, AGVHD, CGVHD, VOC, and stroke. The predicted outcomes are also pictorially represented in pie charts. One unique feature is that the user has the option of selecting a published HCT regimen or selecting a custom regimen by combining conditioning, GVHD prophylaxis, and serotherapy (Figure 3; Table S4 in Multimedia Appendix 9). This feature decreases keystrokes, improves ease of use of the app, and facilitates a comparative analysis across donor types and treatment regimens for different donor types. SPRIGHT can be accessed on any smartphone, tablet, or personal computer using a shortened URL or a QR code. The individualized outcomes are also represented as pie charts displaying individualized estimates (Figure 3). The pie charts can be downloaded and shared with the patient or added to the electronic medical record as an image.

‎

Discussion

Principal Findings

We describe the development and internal validation of SPRIGHT, which, to the best of our knowledge, is the first ML individualized prediction tool for HCT for SCD. Eapen et al [5] identified age, donor type, and conditioning regimen intensity as critical predictive factors of outcomes of HCT for SCD. Gluckman et al [6] identified age and year of transplant as critical factors. Younger patients were shown to have higher EFS. Cappelli et al [43] reported better OS and EFS and a lower incidence of AGVHD and CGVHD in younger patients. Together, these registry-based studies generated important population-level predictive factors of HCT for SCD. They do not, however, provide a means to combine patient, transplant, and disease characteristics into a personalized predictive model for outcomes of HCT. The SPRIGHT prediction model incorporates multiple relevant pre-HCT predictive factors for the individualized production of key clinically relevant post-HCT outcomes. The RF algorithm outperforms the Brazauskas model, other ML algorithms, and logistic regression in predictive performance. The RF-based SPRIGHT prediction model has high predictive discrimination and calibration performance. The excellent discriminative predictive performance is demonstrated by the high value of AUC across all outcomes across all eras, age groups, and follow-up periods of 1 or 3 years. Going beyond the commonly reported predictive discrimination with AUC, we also reported accuracy and balanced accuracy and calibration measures of calibration curve, slope, and intercept. An important innovation of SPRIGHT is the option for the end user to select a published regimen that combines chemotherapy or radiation conditioning, serotherapy, and GVHD prophylaxis. This innovation simplifies the decision-making process for clinicians and allows them to compare potential outcomes across different regimens and donor types. Thus, SPRIGHT helps physicians and patients in discerning the nuances in efficacy and safety of HCT for the individual and has the potential to inform and guide shared decision-making. To mitigate overfitting and validate model performance, we used 10-fold cross-validation, ensuring robustness by mimicking multiple tests on independent datasets. This method approximates the effectiveness of external validation by exposing the model to various training and validation splits, thus predicting its behavior on unseen data. We addressed potential feature collinearity using recursive feature elimination for feature selection and RFs, which inherently mitigate correlation effects through random feature subsampling at each tree.

Limitations

There are several limitations to this study. The SPRIGHT predictive tool is based on the HCT for the SCD dataset derived from data reported to CIBMTR. The use of the CIBMTR dataset allows us to access the entire US experience reported, but it is also subject to limitations of registry studies, including bias, loss to follow-up, and a lack of generalizability across era, center volume, and expertise. While SPRIGHT uses multiple predictive variables, it is limited to those variables collected by CIBMTR. In 2019, Bolanos-Meade et al [44] reported an improved EFS following mismatched relative donor HCT as compared to their previous report in 2009 following the escalation of the dose of total body irradiation from 200 to 400 cGy [6,45]. However, CIBMTR form 2400 and consequently the HCT for the SCD dataset do not distinguish between patients who received the 2 dose levels of total body irradiation. Further, since pain crisis post-HCT is included as a discrete variable in the HCT for the SCD dataset, it is not possible to discern its timing, frequency, or severity. In the CIBMTR dataset, an analysis of predictor completeness reveals that 10/13 TED variables chosen have a completion rate exceeding 98%. However, only approximately 46.5% (763/1641) of patients are on the CRF track, with comprehensive disease-specific data available, a category under which 4 of our predictor variables fall. While the missing data could be a source of bias, the missingness of data was only a function of whether the institution was designated as a TED-only or CRF and whether the CIBMTR algorithm assigned an individual patient to the CRF track. Thus, the missing data may be missing at random. The model we used for imputation, MissForest, has been shown to outperform all other algorithms in all metrics [46,47]. However, these imputation algorithms can produce severely biased regression coefficients and require a careful critique of the missing data mechanism and the interrelationships between the variables in the data [48]. Overall, we acknowledge the concerns regarding the lack of details of SCD-related complications and the completeness of reporting of SCD-related clinical outcomes in the CIBMTR dataset. We, however, also recognize that the CIBMTR registry, with federally mandated data submission, contains the most complete data available. We support ongoing efforts to refine the data collection measures and training of data collection staff. One of the limitations of our model is the rarity of death events in the dataset, which limits the precision of calibration for OS. As a result, the model tends to slightly overestimate OS risk, particularly in subgroups with fewer events. This calibration limitation should be considered when interpreting OS predictions. Across all outcomes, despite mitigatory efforts to combat the inherent bias due to class imbalance, the bias may not be fully eradicated.

The tool incorporates a set of treatment regimens that have been carefully selected from peer-reviewed studies, ensuring that they are backed by sufficient clinical data. These regimens provide reliable and evidence-based predictions. However, custom combinations that are entered by users may fall outside of the dataset’s training and may not be fully supported by the underlying data. In such cases, it is essential to rely on clinical expertise and user discretion when interpreting the predictions.

We recognize a limitation in the registry data due to inherent selection bias, with limited insight into the clinical reasoning behind regimen choices. While the data detail the regimens administered, the underlying clinical judgment is often not recorded. This limitation is particularly notable in HCT, where the absence of universally established standards of care presents additional challenges. Thus, the model and its results should be interpreted within this context.

Comparison With Prior Work

Brazauskas et al [16] developed a risk score using age and donor type as discrete variables. They reported that patients aged ≤12 years with an HLA-matched sibling donor were at the lowest risk. Patients aged ≥13 years with an HLA-matched sibling donor or aged ≤12 years with an HLA-matched unrelated donor were at intermediate risk. All other groups were at high risk. This simple risk score has good predictive performance but has certain gaps that limit its utility in the individualized prediction of outcomes. Gluckman et al [6] used age as a continuous variable and observed that for every 1-year increment in age, there was a 9% increase in the HR for treatment failure (graft failure or death) and a 10% increase in the HR for death. Thus, the Brazauskas model does not include the potential predictive value of increasing age from 5 to 13 years. Further, Brazauskas et al [16] do not incorporate other patient, disease, and conditioning regimen characteristics in the prediction model because they considered these factors to be dynamic and subject to change. The Brazauskas model limits the predicted post-HCT outcomes to death, OS, and EFS and does not include other outcomes that are important to physicians and patients in shared decision-making, such as AGVHD, CGVHD, recurrent pain crisis, or stroke after HCT [13,49-51]. SPRIGHT predicts these outcomes with high predictive performance. Other published ML predictive models for HCT for other diseases are of limited clinical relevance in decision-making since they limit themselves to predicting single outcomes, such as death, overall survival, disease relapse, GVHD, busulfan exposure, kidney injury, or reactivation of Epstein-Barr virus [17-30].

In developing SPRIGHT, we addressed the gaps in the knowledge in individualized prediction of outcomes of HCT. We included 7 clinically relevant outcomes, including rates of OS, EFS, GF, death, AGVHD, CGVHD, and VOC. Each of these 7 clinical outcomes initially required a distinct set of 10‐11 pre-HCT predictive features for optimal performance, leading to incomplete overlap and potential model fragmentation. For addressing this, we adopted a unified approach by selecting a comprehensive set of 17 pre-HCT patient, HCT, and disease characteristics. By applying robust feature selection techniques to optimize predictive performance and improve the model’s overall clinical applicability, we demonstrated the predictive value of these features. Further, patient and disease features inform patient selection and HCT features inform regimen selection. Thus, these additional pre-HCT features are important, clinically relevant considerations in decision-making.

Future Directions

We present here an initial in-lab validation of the predictive model. We recognize that the use of a US-based dataset may limit the generalizability of our findings to other regions or health care systems due to potential differences in population characteristics and treatment protocols. For further external validation across different geographic locations to establish the model’s generalizability and clinical utility, we propose to use the European Bone Marrow Transplantation Registry (EBMT) dataset, a completely independent dataset, in collaboration with European investigators. Of note, Gluckman et al [6] have previously combined CIBMTR and EBMT registry data on SCD for analysis and included similar numbers of children, adults, and donor types from the 2 registries and do not report differences in predictive factors or outcomes in the 2 registries. We have demonstrated that the predictive performance remains equivalent whether we use the entire dataset or the more recent data after 2007, which represents 84% (1378/1641) of participants in the dataset. Acknowledging the evolving nature of supportive care practices in HCT, we propose further temporal validation with future years of data being added to this dataset. We also propose to continue to enhance SPRIGHT by incorporating expert opinion, adapting to patient health literacy, values, and preferences, and using patient-friendly data visualization to support shared decision-making [52-54].

Conclusions

In conclusion, the SPRIGHT prediction model integrates individual-specific patient and disease characteristics, conditioning regimens, GVHD prophylaxis, and donor characteristics and predicts key clinical outcomes. It exhibits superior predictive performance across multiple measures of discrimination and calibration as compared to logistic regression and other ensemble ML methods.