Facilitator messages from session transcripts were coded independently by trained human coders and an AI-based coder using a common MI coding frame. Agreement was examined at two levels: (1) at the level of facilitator messages for the identification of specific MI techniques and (2) at the session level, for assessing implementation adherence across intervention stages. Reporting follows the GRRAS (Guidelines for Reporting Reliability and Agreement Studies) for the intercoder agreement components [18] and is informed by the TRIPOD-LLM (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis–Large Language Models) reporting guidance for studies that develop or evaluate LLMs [19]; completed checklists for both guidelines are provided Checklists 1 and 2.

The Brief Intervention Based on Motivational Interviewing program (Intervención Breve Basada en Entrevista Motivacional [IBEM]) served as the exemplar intervention for this study. IBEM is a brief, school-based preventive intervention, developed in Colombia, and grounded in the principles of MI. Its goal is to support adolescents in reflecting on their motivations and behaviors related to substance use, with the aim of promoting healthier decision-making and reducing risk-related behaviors. The intervention consists of a single core session followed by 2 follow-up meetings. The initial session, lasting approximately 15 to 20 minutes, centers on a personalized MI-guided conversation about alcohol and other substance use. Facilitators conduct a brief assessment; provide individualized feedback; and guide adolescents in exploring their motivations, perceived risks, and readiness for change. Through this dialogue, adolescents identify self-directed goals and develop strategies to reduce or prevent substance use. The 2 follow-up sessions are conducted at 3 and 6 months and focus on reviewing progress toward previously established goals, discussing achievements, and identifying barriers and facilitators that may influence behavior change. IBEM has been developed in Colombia and was implemented also in Mexico and Brazil. Evaluations of IBEM in Colombia have shown the intervention to be efficacious in reducing adolescent alcohol use and its associated risks [20,21]. In 2021, IBEM received the National Award for Best Practices in Prevention, granted by the Colombian Ministry of Health and Social Protection in recognition of its contribution to prevention practice.

Two expert human coders were recruited in consultation with IBEM program developers [22] and were invited to participate in the study based on their prior certification in the intervention and demonstrated experience implementing it in school-based settings. Both coders were Colombian professionals and native Spanish speakers, reflecting the linguistic and cultural context in which IBEM was originally developed and delivered. Each coder held a professional degree in psychology and postgraduate-level training and completed formal instruction in MI. IBEM certification required a 32-hour training in adolescent substance use (ie, alcohol, tobacco, and other illicit drug use), MI principles and techniques, role-playing, and implementation evaluation. Both coders possessed over 5 years of experience delivering IBEM in real-world implementation contexts.

Two transcripts were selected from a pool of 10 IBEM sessions conducted in Spanish by certified implementers, originally collected as part of a previous research project focused on developing an AI chatbot to simulate adolescent responses during IBEM delivery. IBEM sessions, therefore, were not routine clinical encounters; certified expert facilitators delivered complete IBEM sessions while interacting with a simulated adolescent, which allowed facilitators to exercise the full MI protocol under controlled conditions. The selection of IBEM sessions was based on three considerations: (1) expert review indicated that the sessions reflected typical IBEM delivery rather than either flawless delivery or substantial omission of protocol components, (2) sessions were of average length relative to the pool of sessions, and (3) coders were not assigned to transcripts from sessions they had personally facilitated. A facilitator message was defined as a single, uninterrupted facilitator turn in the transcript; when a turn contained several sentences, it was treated as 1 message and could receive multiple technique codes. The selected transcripts yielded 72 facilitator messages (32 messages from session 1 and 40 messages from session 2).

Human coders completed a structured virtual training based on the MI coding manual prior to transcript review. Coding involved 2 distinct tasks. First, coders evaluated MI techniques at the facilitator-message level, assigning codes to each individual facilitator message. Second, coders assessed intervention adherence at the item by session level using the 21-item checklist of required stages of IBEM implementation. After independently coding the first transcript, each human coder met individually with the research team to clarify ambiguities in the coding process. Minor refinements to category definitions were incorporated before the coding of the second transcript began. Coders were blinded to one another’s ratings and the AI-generated codes throughout the process. AI codes were generated in parallel and merged with the human-coded dataset for comparative analyses.

Intercoder agreement for MI techniques was assessed using Cohen κ [29], calculated separately for each technique category and pair of coders (coder 1 vs coder 2, coder 1 vs AI coder, and coder 2 vs AI coder). To examine whether coders differed systematically in their likelihood of identifying techniques for the same messages, Cochran Q tests were conducted for each technique across all 3 coders. Descriptive statistics were used to summarize implementation adherence ratings by item and session (ie, 21 items × 2 sessions = 42 ratings for each coder). Overall intercoder agreement among the 3 coders was estimated using Fleiss κ [30]. Pairwise intercoder agreement (coder 1 vs coder 2, coder 1 vs AI coder, and coder 2 vs AI coder) was evaluated using Cohen κ with 95% CIs. In addition, percentage agreement between pairs of coders was calculated to facilitate the interpretation of intercoder agreement statistics. Following standard practice, κ values were interpreted using the benchmarks proposed by Landis and Koch [31], specified a priori, in which values of 0.00 to 0.20 indicate slight agreement, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, and 0.81 to 1.00 almost perfect agreement.

All procedures involving human participants were reviewed and approved by the University of Miami Institutional Review Board (ID: 20250114; approval date: March 7, 2025). The study was conducted in accordance with the ethical standards of the institutional research committee. Informed consent was obtained through 2 separate processes: facilitators who participated in the original IBEM sessions provided consent to share the transcripts of their sessions. Prior to any analysis, all transcripts were deidentified by removing names and other direct identifiers so that messages submitted for coding did not contain personally identifying information. Deidentified facilitator messages were transmitted to OpenAI’s GPT 4.1 through the API solely for classification; under the API terms in effect at the time of the study, submitted data were not used to train or improve OpenAI models.

Across the 72 facilitator messages, open questions represented the most frequently coded MI technique across the 2 human coders consisting of 29% (n=21) and 31% (n=22) of the messages, respectively. Giving information was identified in 19% (n=14) of the messages by both coders. Affirmations were coded in 18% (n=13) and 15% (n=11) of the messages, and closed questions were coded in 17% (n=12) and 18% (n=13) of the messages, respectively. Reflective statements occurred less frequently; simple reflections were identified in 6% (n=4) and 4% (n=3) of the messages, and complex reflections were identified in 6% (n=4) and 4% (n=3) of the messages, respectively. Confrontation was identified in one (1%) instance by coder 2 and not by coder 1. Messages not otherwise categorized accounted for 18% (n=13) and 21% (n=15) of the messages. The AI-based coder demonstrated a generally similar coding distribution of facilitator messages. Open questions were identified in 28% (n=20) of the messages, followed by closed questions (n=16, 22%), affirmations (n=14, 19%), and giving information (n=14, 19%). Simple reflections were identified in 3% (n=2) of the messages, and no complex reflections were detected. Messages coded as not categorized comprised 17% (n=12) of the total messages.

Cochran Q tests revealed no statistically significant differences in the percentages of MI technique use across the 3 coders for any category: affirmations, Q(2, n=72)=1.56, P=.46; not categorized, Q(2, n=72)=1.56, P=.46; open questions, Q(2, n=72)=0.43, P=.81; closed questions, Q(2, n=72)=1.73, P=.42; giving information, Q(2, n=72)=0.00, P>.99; simple reflections, Q(2, n=72)=1.20, P=.55; confrontation, Q(2, n=72)=2.00, P=.37; and complex reflections, Q(2, n=72)=5.20, P=.07. The unclear category was not evaluated because it was not endorsed by any coder. The distributions of MI techniques by coder and session are presented in Figure 1.

Between the 2 human coders, agreement was high for giving information (κ=0.91) and substantial for affirmations, open questions, closed questions, and messages not categorized (κ values ranging from 0.66 to 0.82). Agreement was lower for complex reflections (κ=0.55) and lowest for simple reflections (κ=0.25). The AI-based coder showed a similar pattern of agreement with both human coders. For the most frequently coded categories (giving information, affirmations, and open questions), AI-human agreement was substantial to high (κ =0.66-0.91). The main discrepancies involved low-frequency categories: the AI coder did not identify any complex reflections, resulting in κ of 0.00 for both AI-human pairs, and agreement on simple reflections was inconsistent across pairs (κ=0.31 with coder 1 vs κ=0.79 with coder 2). Agreement on closed questions was moderate for the AI-human pairs (κ=0.47 and 0.53), compared with substantial agreement between the two human coders (κ=0.66). Cohen κ coefficients with 95% CIs for all coder pairs are presented in Table 1.

Across the 2 IBEM sessions, adherence to implementation stages varied by session but showed substantial overlap among AI and human coders. Using the 21-item checklist, coders 1 and 2 endorsed 12 (57.1%) and 10 (47.6%) items in session 1, respectively, whereas the AI coder endorsed 11 (52.4%). In session 2, endorsement rates were higher for all coders (coder 1: n=16, 76.2%; coder 2: n=17, 81.0%; AI coder: n=20, 95.2%). When checklist ratings were pooled between sessions (42 ratings), full 3-way agreement occurred for 28 (66.7%) of the items. In terms of pairwise percentage agreement, concordance was 85.7% between the AI coder and coder 2, 78.6% between the 2 human coders, and 69.0% between the AI coder and coder 1. Disagreements clustered primarily in stage 2 (Risk Assessment) and stage 4 (Importance and Confidence), whereas stage 1 (Presentation and Contextualization) showed near-uniform agreement across coders. Overall, interrater agreement across the 3 coders, assessed with Fleiss κ, was moderate (κ=0.487, z=5.47; P<.001). Pairwise Cohen κ indicated moderate agreement between the 2 human coders (κ=0.526, 95% CI 0.256-0.797) and substantial agreement between coder 2 and the AI coder (κ=0.669, 95% CI 0.432-0.907), whereas agreement between coder 1 and the AI coder was fair (κ=0.264, 95% CI 0.043-0.571). Figure 2 presents agreement patterns for each adherence item by the stage of IBEM implementation across the 2 IBEM sessions.

For each of the 21 checklist items in each of the 2 sessions, “agreement” denotes concordance among coders on the same present (1) or absent (0) rating. The figure displays both 3-way agreement (all 3 coders assigning the same rating) and pairwise agreement, organized by the 6 IBEM implementation stages.

This study examined the use of an LLM to automate the identification of intervention techniques and implementation adherence coding within the context of a brief MI-based preventive intervention for adolescents. Agreement between the AI-based coder and trained human coders was assessed at 2 levels: individual facilitator messages for MI technique identification and item-level checklists for session-specific implementation adherence. Using the framework of Carroll et al [17], these may be thought of as implementation quality of delivery and adherence, 2 core components of implementation fidelity. Proctor et al [32] position fidelity as a key implementation outcome that mediates the relationship between implementation strategies and intervention effectiveness, underscoring the importance of scalable fidelity measurement. Overall, our findings provide support for the feasibility of AI-assisted coding of MI techniques and implementation adherence, although they reveal specific limitations that require further investigation. These results were derived from a proof-of-concept dataset and were therefore intended to establish initial feasibility rather than to justify large-scale deployment or to guide consequential coding decisions in practice. Accordingly, the estimates reported below should be interpreted with caution.

Consistent with our first hypothesis, the AI-based coder demonstrated moderate-to-substantial agreement with human coders across the most commonly used MI techniques. Agreement was particularly strong for giving information, affirmations, and open questions, techniques characterized by relatively explicit linguistic markers that may be easier for automated detection [10]. Importantly, Cochran Q tests revealed no statistically significant differences in the percentages of MI technique use across the 3 coders for any technique category, supporting our hypothesis that the AI system would not exhibit systematic bias in category assignment. These agreement levels are broadly consistent with those reported in prior automated MI coding research. For example, Tanana et al [9] found that both deep-learning models achieved high agreement with human coders for open and closed questions, giving information, and affirmations (all κ>.50), with modestly lower agreement for simple and complex reflections (κ values between 0.30 and 0.50). The present study, using a general-purpose LLM without task-specific training, achieved comparable or higher agreement for these high-frequency categories, suggesting that advances in language modeling may reduce the need for domain-specific model training.

We note, however, that the AI coder’s performance on reflection statements was notably weaker than other MI techniques. The AI system failed to identify any complex reflections, producing κ values of 0.00 for both human-AI comparisons. Complex reflections require inferring implicit meaning, emotional undertones, and rephrasing that extends beyond the literal content of participant statements, processes that demand a level of pragmatic interpretation that current LLMs struggle with [13]. Notably, complex reflections were also among the least frequently coded categories by human raters, and agreement between the 2 human coders on this category was only moderate (κ=0.55), suggesting that these distinctions are inherently difficult even for trained experts. For simple reflections, an interesting asymmetry emerged: agreement between the AI and coder 2 was substantial (κ=0.79), whereas agreement with coder 1 was fair (κ=0.31). This variability likely reflects individual differences in how coders operationalize the boundary between simple reflections and similar categories such as closed questions or paraphrases, a well-documented challenge in MI coding [33], and suggests that coding MI techniques is a difficult task even for human experts. The AI coder’s inability to detect complex reflections echoes findings from earlier NLP-based approaches. Tanana et al [9] similarly reported lower accuracy for reflections compared with questions, and Atkins et al [10] noted that categories requiring pragmatic inference have the greatest challenge for statistical classifiers. The present findings suggest that this limitation persists even with more advanced language models.

The results from the IBEM 21-item implementation adherence checklist were similarly promising. Full 3-way agreement occurred for two-thirds of the checklist items (n=28, 66.7%), and the AI coder achieved the highest pairwise percentage agreement with coder 2 (n=36, 85.7%). Pairwise κ between the AI coder and coder 2 was substantial (κ=0.66), exceeding the moderate agreement observed between the 2 human coders themselves (κ=0.52). These findings suggest that the AI coder was able to interpret session structure and procedural adherence with a degree of reliability comparable to human-to-human agreement. Checklists are commonly used in implementation settings to monitor facilitator compliance with program protocols [34], and the present results indicate that AI-based monitoring could meaningfully support this function and speed up the pace of adherence monitoring.

That said, agreement between the AI coder and Coder 1 was only fair (κ=0.264), indicating unreliable concordance for this pair. This discrepancy, however, should be interpreted in light of the disagreement among the human coders themselves rather than as evidence of a problem unique to the AI coder. The 2 human coders agreed only moderately on adherence (κ=0.52), so there was no single human “gold standard” against which the AI coder could be benchmarked. Critically, the AI coder aligned substantially with coder 2 (κ=0.66) but only fairly with coder 1, and coder 1 also diverged from coder 2 to a comparable degree. Coder 1 was therefore the most distinct rater of the three, and the AI coder’s weaker agreement with that coder largely mirrors a preexisting difference between the 2 humans.

Disagreements clustered in stage 2 (Risk Assessment) and stage 4 (Importance and Confidence), suggesting that adherence checklist items requiring interpretation of facilitator intent, rather than the presence of explicit content, remain challenging for both human and AI coders. It is also worth noting that checklist-based adherence assessment has inherent limitations: sessions are highly sensitive to the context of each interaction. For instance, some adolescents may be more engaged or collaborative, while others may be more reserved, leading facilitators to adapt their delivery accordingly. These contextual variations may produce natural differences in how facilitators approach each stage, which in turn complicates the interpretation of adherence ratings regardless of whether the coder is human or AI.

From an implementation science perspective, the AI coder’s ability to consistently recognize most core MI techniques and procedural steps represents a meaningful advancement for scalable fidelity monitoring. Returning to the conceptualization of implementation fidelity of Carroll et al [17], the AI coder performed well on adherence measurement, where agreement with one human coder exceeded human-to-human agreement, and on identification of MI techniques (a potential indicator of quality of intervention delivery) for categories with explicit linguistic markers. This pattern suggests that LLM-based coding may be most immediately viable for the adherence dimension of fidelity, which relies on detecting the presence or absence of prescribed intervention components, while the quality of delivery dimension of fidelity requires further development for higher-inference techniques such as complex reflections. Manual coding of MI sessions is time-intensive and costly, and its reliance on trained human coders limits the frequency of fidelity monitoring. An AI-assisted approach could reduce human workload, increase the feasibility of conducting routine fidelity checks, and enable more timely feedback to facilitators, particularly in resource-constrained or geographically dispersed implementation settings. In practical terms, these results point toward the need for a human-in-the-loop workflow rather than full automation. In such a model, the AI coder would “precode” facilitator messages and adherence items and flag low-confidence or higher-inference cases, such as complex reflections and the stage 2 and stage 4 items where agreement was the weakest.

Finally, a human-in-the-loop workflow should treat the AI coder as an annotation instrument and assess whether its outputs systematically change the distribution of codes across categories. For example, Xu et al [35] found that an LLM annotator shifted the overall distribution of emotional-tone labels toward the “very negative” category. In our study, the AI coder identified no complex reflections, whereas the human coders identified some examples of this higher-inference technique. Although these counts are too small to establish systematic undercoding, they illustrate why human oversight should examine whether the AI consistently over- or underrepresents specific categories before its codes are used in fidelity monitoring or as training labels.

Several limitations should be considered when interpreting these findings. First, the study analyzed only 2 session transcripts comprising 72 facilitator messages and 42 adherence checklist items, which limits the generalizability of the study findings. With the small number of observations per category, the agreement estimates have limited precision, as reflected in the wide CIs, and the results should be regarded as preliminary evidence. Second, only 2 human coders participated, which restricts the estimation of human-to-human reliability and does not capture the full range of variability that would be observed in a larger coder panel. Because human-to-human reliability was itself moderate, comparisons between the AI coder and each individual coder were sensitive to that coder’s idiosyncratic coding style. A larger and more diverse panel of coders would provide a more stable human reference standard against which to benchmark the AI coder. Third, the study evaluated a single LLM (GPT 4.1) with a single prompt configuration; performance may differ across models, prompt versions, and prompt engineering strategies, and the study results should not be generalized to other LLMs without direct comparison. Fourth, all transcripts were in Spanish from a Colombian intervention context, and it remains unknown whether agreement levels would hold for MI sessions conducted in other languages or cultural settings. Relatedly, the transcripts were drawn from standardized facilitator practice sessions in which the interlocutor was a chatbot simulating adolescent responses rather than a live client; although this design isolated facilitator MI behaviors under controlled conditions, conversational dynamics may differ from naturalistic school-based sessions, and the findings should be replicated with live adolescent interactions before broader use. Finally, the coding framework used in this study focused on behavioral counts of MI techniques and binary adherence checklist items; the study did not evaluate higher-order global competencies such as empathy, which may present additional challenges for automated classification.

Future research should address these limitations by (1) expanding the number and diversity of transcripts to include sessions from multiple intervention sites and facilitators, (2) comparing the performance of multiple LLMs and systematically evaluating the impact of prompt engineering on classification accuracy, (3) examining whether AI-assisted feedback can improve both proximal (eg, facilitator adherence) and distal (eg, program sustainability) implementation outcomes in prospective designs, and (4) extending the coding framework to include global MI competencies (eg, empathy) to test automated coding of complex MI techniques. Additionally, cross-linguistic validation studies are needed to determine whether the present findings generalize to MI sessions conducted in English and other languages. Integrating AI-based coding into real-time supervision workflows represents a particularly promising direction for translating these findings into implementation practice.