Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) approach, a search on Scopus and PubMed electronic databases was conducted. This search aimed to identify peer-reviewed English conference and journal papers published between January 2011 and August 2022. Scopus indexes a larger number of peer-reviewed scientific journals than the Web of Science and offers results of more consistent accuracy than Google Scholar, while PubMed remains a leading database in biomedical research [24]. Including papers published within the past decade will reveal recent evidence-based research trends [25]. Using the logical OR/AND operators, the search phrases were a combination of keywords related to prediction, adherence, health behavior change interventions, and machine learning (See Multimedia Appendix 1 for the search phrases). Considering the plethora of approaches to investigate adherence, the search for eligible studies was not limited to a specific study design.

Only empirical studies that described the development and testing of machine learning models for BCSS adherence prediction were considered. Studies that used only self-reported data, were not reported in English, or did not focus on human participants were ignored.

During the initial search of the databases, 2182 papers were retrieved. The results were refined by year, document type, publication stage, source type, and language, resulting in 1866 papers. The exported papers were screened for uniqueness and for titles containing keywords such as adherence, prediction, and any machine learning technique. Of these, 1812 papers were excluded. The remaining 54 papers were screened by abstracts and full texts. Papers were excluded by abstract if the machine learning technique(s) were not mentioned. Furthermore, papers were excluded by full text if the intervention was not characterized by a BCSS (any form of information system that has been developed to change human behavior voluntarily). Finally, 11 journal papers were considered eligible for this systematic review and thus downloaded for methodological quality assessment.

Figure 1 shows the PRISMA flow diagram of study identification and selection. Since the study selection was not limited to a specific study design, the Mixed Methods Appraisal Tool by [26] was used to assess the methodological quality of the downloaded papers (see Multimedia Appendix 2 [13,14,26-35]). Accordingly, 11 studies were identified to be of high methodological quality. Pertinent information on the study characteristics and machine learning approaches was extracted using a data extraction form in Microsoft Excel created by the authors. Multimedia Appendix 3 [13,14,27-35] presents a list of included studies.

The BCSSs included mobile apps [13,14,27,28,33,34], web-based apps [30,32,33,35], and sensor-based systems plus mobile apps [29,31]. They were targeted at different groups of people, namely physically inactive women, illicit drug users, obese or overweight people, and patients with a wide range of chronic diseases, such as heart failure, myocardial infarction-anxiety, depression (MI-ANXDEP), insomnia, and Parkinson disease. Multimedia Appendix 4 [14,15,29-44] describes other study-specific characteristics.

The BCSSs leveraged some behavior change techniques and persuasive systems features to improve user adherence. These features were extracted and evaluated using the persuasive systems design (PSD) model [45]. The PSD model has been validated in several studies [37] and is predominately used in the design and evaluation of BCSSs [46]. The model consists of 28 system features that make up 4 categories of persuasive principles (namely primary task support, dialogue support, credibility support, and social support). Table 2 displays the frequency of the PSD features represented in the BCSS. All the studies had primary task support features, 8 studies had dialogue support features, 5 studies had credibility support features, and only 1 study had social support features.

Primary task support simplifies and motivates users to perform recommended tasks (eg, exercise). A total of 5 features of the primary task support principle were used: personalization, self-monitoring, reduction, rehearsals, and tailoring. Personalization was the most used feature in this category. It delivers personalized content to the users. For example, artificial intelligence generated workouts according to an individual’s characteristics and preferences. Self-monitoring enables app users to view and track their activity levels and health status in real-time (eg, the app enables users to monitor, visualize, and track activity levels and calories burned in real time). Reduction breaks down tasks such as daily point goals into specific meal or snack targets. The least used features in this category were rehearsal (practicing the target behavior, eg, gait movements) and tailoring (eg, the app provided content that was distinct to users of specific age groups and health goals, such as alcohol or smoking cessation, weight loss, or mental health).

Dialogue support provides a means to help users to achieve their goals via human-computer interactions. The dialogue support features included reminders (eg, medication prompts), suggestions (eg, the app advises users based on their input to the app), and praise (automated feedback on the completion of a task). The least used features included rewards (eg, point-based incentives), similarity (eg, therapy resembling traditional cognitive behavioral therapy), and liking (eg, user-friendly and appealing design).

System credibility support provides a means for users to trust the system. Features identified in this category included expertise (app provided theory-based information and were designed to improve engagement, effectiveness, and security), verifiability (app provided links to related sites), third-party endorsement (from the National Institute of Health), and real-world feel. The real-world feel feature was implemented as “Contact Us” (a means to communicate with the developers of the app) and “About Us” (providing information about the developers of the app).

Social support provides a means of supporting users via social influence. However, it was the least used principle. Social facilitation was the only identified social support feature and it was implemented by allowing user participation in online app forums. Perhaps, the minimal use of social support features may be attributed to the negative sentiments associated with some of its features [38].

It is important to note that the effectiveness of these persuasive system features in improving adherence was not explicitly evaluated in the included studies. However, these features may have directly or indirectly improved adherence rates. Some of the studies used behavior change techniques such as goal-setting, web-based human coaching, face-to-face counseling sessions, and feedback from psychologists and expert program providers to improve adherence behavior. However, this review could not identify behavioral theories or models upon which these techniques were based.

Developing machine learning models for predicting user adherence or nonadherence was the general aim of all the included studies. This process conventionally consists of 4 main stages: data collection, feature selection, model training, and model validation.

Apart from the gait-related data, which were collected in a controlled environment (laboratory), the data used to train or test the machine learning models in the majority of the studies were objectively collected by the health app while the study participants were performing the behavior of interest in an uncontrolled environment. For example, data such as log data, training behavior, walking steps, and 3D movement scans were automatically extracted in a contactless manner without any form of self-report from the users.

Studies on medication adherence were observed to use images and videos to capture participants’ medication adherence behavior. The apps used technologies such as computer vision [34], internet-connected smart sharp bin [29], and flight sensors [31] for the real-time monitoring of self-administered injections and medication ingestion. Time-stamped data of injection needles discarded into the smart sharp bin, time-stamped skeletal joint data, and images of the participants taking the medication were retrieved, validated, and then used to generate the data set for training or testing the model. Similarly, studies on physical activity adherence [14,27] used continuously collected time-dependent physical activity data from app users to develop machine learning models.

Studies on adherence to digital cognitive or behavioral interventions [13,28,30,32,35] and dietary adherence [33] used a combination of objectively collected data and participants’ responses to questions on self-assessment delivered by the health app. Goldstein et al [33] used a BCSS that asked predefined questions related to triggers of dietary relapse for the analysis. Considering that objectively collecting trigger-related data or self-assessment data on health symptoms from a mobile app may currently be challenging as triggers (such as food cravings and hunger) are physiologically motivated, future studies on BCSSs that seek to extract trigger-related data may consider using physiological sensor data. This is a noninvasive approach to detecting hunger and cravings using wearable body sensors [39]. Such sensor data may also be integrated into the health app to enable self-monitoring.

Among the 11 studies, 5 performed feature selection, 5 performed feature engineering, and 1 adopted features based on existing literature (see Multimedia Appendix 5 [14,15,29-37]). Feature selection and feature engineering were both aimed at enhancing model performance by eliminating irrelevant features and generating new features from raw data respectively. Due to the complexities of combining 2 or more machine learning techniques (ie, ensemble learning), some studies [28] applied more than 1 feature selection method. However, there were no differences in the selected features.

The flat features algorithm (including filter, wrapper, and embedded methods) were the predominately used feature selection method. This algorithm assumes all features to be independent [40]. Interestingly, each study had its own set of unique predictors irrespective of the category of adherence problem. Multimedia Appendix 5 [14,15,29-37] highlights the various feature selection approaches.

The learning problem was a binary classification. Thus, there were 2 class labels (outcomes), namely adherence/nonadherence, adherers/nonadherers, relapse/nonrelapse, and dropout/nondropout. An overview of the adopted techniques and the outcomes of the best-performing techniques is provided in Table 3.

A total of 13 supervised machine-learning techniques were used across the included studies. Logistic regression, support vector machines, and random forest were the most used techniques cutting across all 4 categories of the adherence problems. The machine learning techniques mapped to specific adherence problems included support vector machines for physical activity adherence; extreme gradient boosting, extra trees, recurrent neural network, naive Bayes, and gradient tree boosting for medication adherence; and ensemble methods for dietary adherence. Random forest was observed to be the predominant best-performing model in studies on adherence to digital cognitive or behavioral interventions, while long short-term memory (LSTM) was a common best-performing model between medication adherence and physical activity adherence. Overall, the predominant best-performing models across all included studies were random forest, decision trees, logistic regression, k-nearest neighbor, LSTM, and ensemble learning.

Owing to the relatively small and imbalanced data sets used in some of the included studies, cross-validation methods were adopted to eliminate bias that may occur during data split. The following cross-validation methods were identified: K(5)-fold cross-validation [29,34]; leave-one-out cross-validation [28,31,33]; and stratified K(10)-fold cross-validation [13,30,35].

Besides cross-validation methods, several performance metrics were used to compare and evaluate the performance of the various machine learning models. It was observed that the choice of performance metrics was dependent on the context of the study and more than 1 metric was used to evaluate the performance of a model (see Table 3). The predominately used metrics in order of frequency included area under the receiver operating characteristic curve (7/11), accuracy (6/11), sensitivity (6/11), specificity (4/11), precision (3/11), F₁-score (3/11), true positive rate (3/11), false positive rate (3/11), confusion matrix (3/11), positive predictive value (1/11), negative predictive value (1/11), and precision-recall curve (1/11).

Generally, it was observed that the prediction models had good classification accuracies. This was an indication that the features or predictors used in each of these studies were a good representation of the intervention domain. Nonetheless, due to the plethora of digital platforms, the interaction between technology and behavior may affect the generalizability of the results [33,34]. Thus, Koesmahargyo et al [34] posit replication and integration of data from various digital platforms.

Studies suggest that the rate of adherence may be affected by the following:

These barriers may be classified into two nonadherent groups: intentional nonadherence (1 and 2), or unintentional nonadherence (3, 4, and 5).

Findings from this systematic review indicate that though the use of machine learning techniques in the prediction of adherence to BCSSs is scarce and is only beginning to gain research interest, it has the potential to accurately predict adherence behavior in real time using objectively collected data. This systematic review is unique as it has not yet been reported in the literature, and it provides an overview of machine learning approaches in determining predictors of specific adherence problems in BCSSs. A grasp of these trends across different BCSSs will guide researchers in choosing appropriate features and machine learning techniques that favor the prediction of specific adherence problems. In summarizing findings from 11 journal papers, this systematic review highlights research gaps and areas for future research. It also acknowledges limitations that may exist due to the selection strategy for eligible studies.