We are in the middle of a global mental health crisis among young people [1-3]. Depression, anxiety, and related disorders are among the leading causes of illness and disability in people aged 10 to 24 [4], and their prevalence has risen sharply over the past 3 decades [2]. An estimated 279 million young people aged 10-24 years live with a mental health disorder [2,5]. Most receive no help. The treatment gap—the proportion of people who need care but do not receive it—is 40%-60% globally and exceeds 80% in low-resource settings, where the majority of the world’s young people live [6,7].

The core structural problem driving this gap is a global scarcity of providers [1,8,9]. The global median is 13.5 specialized mental health workers per 100,000 people [8]; the ratio falls to just 1.1‐2.4 per 100,000 in low- and middle-income countries, against 67.2 in high-income countries (HICs) [8,9]. Kenya, for instance, where 1 in 3 adolescents experience a mental health problem [10-13], has about 2 specialists per 100,000 people, most of them concentrated in urban centers [9,14]. Closing the treatment gap through specialist-led care alone is not feasible in the near term [15,16].

Task-shifting—training non-specialists to deliver structured, evidence-based interventions under supervision—has emerged as the leading strategy for expanding the workforce and closing the treatment gap [17-19]. Randomized trials across sub-Saharan Africa and South Asia demonstrate that lay providers—including community health workers [20], trained youth [21], and even grandmothers [22]—can deliver effective treatments for depression, anxiety, and trauma, producing clinically meaningful improvements that are sometimes comparable to those achieved by professionals [18,19,23].

The Shamiri Model in Kenya illustrates what task-shifting looks like in practice. Shamiri is a brief, 4-week group intervention for secondary school students built around strengths-based psychological principles—growth mindset, gratitude, and values affirmation [21,24]. It operates through a 3-tier delivery model. In the first tier, lay providers aged 18-22 years are trained to deliver weekly group sessions to cohorts of 6 to 15 students [21,25]. Clinical supervisors with backgrounds in psychology or social work form the second tier, overseeing lay providers and providing direct one-on-one support where needed [21,25]. Complex or severe cases are escalated to licensed clinicians in the third tier [26]. The structure maximizes reach by placing trained lay providers at the point of care; the model operates at the school level—where adolescents spend most of their time—without drawing on scarce specialists for routine cases [26,27]. Randomized controlled trials have demonstrated that the model significantly reduces depression and anxiety symptoms in secondary school students, with effects sustained at least 7 months postintervention [21,28,29].

Task-shifting has demonstrated that an expanded workforce can deliver effective care [18,23,30]. The harder problem is maintaining quality across that workforce as programs grow. Training a cohort of 20 lay providers is manageable; supervising hundreds of them, across dozens of sites, delivering sessions week after week, is a fundamentally different challenge—and one that existing models have not solved [25,26].

The answer, in every evidence-based framework for task-shifting, is supervision [17,25,31]. Supervision is the mechanism through which lay providers consolidate skills, receive corrective feedback, maintain fidelity to protocol, manage clinical risk, and sustain the psychological safety required to work with vulnerable youth [17,32-34]. Without it, skill drift is invisible—sessions degrade in quality, required elements go undelivered, and program managers have no way of knowing [17,25,33,34].

But high-quality supervision is difficult to scale. Traditional supervision models require trained professionals to manually review session recordings, meet individually or in groups with providers, and deliver timely, specific feedback [25,26,33]. As programs grow, supervision becomes a bottleneck. Supervisors can review only a fraction of sessions, feedback arrives late, and important behavioral information—tone, pacing, and interpersonal dynamics—is lost [25,35]. Supervision remains a central constraint on the quality and scalability of task-shifted mental health care.

Two advances offer a route forward. The first is the maturation of task-shifting models themselves. Models like Shamiri have shown that lay-delivered interventions can be implemented at scale with structured training and supervision systems—creating the operational infrastructure into which quality-assurance tools can be embedded [18,19,23,26,30]. The second is the emergence of AI tools capable of automating the observational and feedback functions of supervision. Advances in automatic speech recognition (ASR), natural language processing, and large language models (LLMs) have made it technically feasible to process session audio, estimate provider behaviors, score fidelity to clinical protocols, and generate structured feedback reports—potentially reducing the manual-review bottleneck through systematic, scalable monitoring [36-39]. Research teams in HICs have developed AI systems that achieve human-level reliability on session-quality ratings for individual cognitive behavioral therapy (CBT) in English [36,37,39], and recent work demonstrates that LLMs can score therapeutic constructs from transcribed sessions with strong psychometric properties and valid associations with outcomes [40]. The trajectory suggests AI-assisted fidelity monitoring is becoming viable—but viable under a specific set of conditions: individual therapy, adult populations, professional providers, and English-language settings in HICs.

Whether AI fidelity monitoring can be extended to the conditions that define most of the world’s task-shifted mental health care is entirely unestablished. Group delivery, lay providers, multilingual code-switched speech, minimal training data, and resource-constrained deployment contexts are not peripheral variations—they are the structural features of the settings where supervision is most needed and least available. To our knowledge, no automated fidelity-monitoring system has been developed for, or validated in, this combination of lay-delivered, multilingual, group-format care. The gap spans every axis that matters, including delivery format, workforce type, linguistic environment, and implementation constraints.

To address these gaps, we developed shamiriAI—an AI system designed to support fidelity monitoring and supervision for lay-delivered mental health interventions in multilingual, low-resource settings. shamiriAI processes raw session audio, performs multilingual ASR across English, Kiswahili, and Sheng, extracts prosodic features capturing group dynamics and facilitator behavior, and generates structured written feedback reports for clinical supervisors. It operates in an AI-in-the-loop configuration: supervisors remain the primary agents of support and clinical judgment; AI-generated feedback extends their observational reach, providing structured input on sessions that would otherwise go unreviewed.

This study reports the foundational validation work required before AI-augmented supervision can be meaningfully tested for its effect on provider skill or student outcomes. Before asking whether the system works clinically, 2 prior questions must be answered: can it accurately transcribe sessions in this linguistic environment, and are its fidelity assessments sufficiently aligned with expert human judgment to constitute credible supervisory inputs?

The objectives of this pilot study were twofold. The first was to evaluate the performance of shamiriAI’s multilingual ASR pipeline on a held-out test set of manually transcribed sessions, using metrics appropriate for code-switched, agglutinative speech. The second was to assess interrater reliability between shamiriAI-generated fidelity ratings and independent human supervisor ratings across 6 fidelity dimensions, examining the degree of agreement, the direction and magnitude of systematic bias, and the specific dimensions for which the AI most and least closely approximated expert human judgment.

These aims are explicitly foundational. They establish whether shamiriAI produces transcripts and fidelity ratings with sufficient accuracy to support future development—the essential first step toward an AI quality assurance infrastructure capable of supporting the supervision of lay providers at population scale across sub-Saharan Africa.

We conducted a pilot validation study of shamiriAI, an AI-based supervision support tool, embedded within routine delivery of the Shamiri intervention across 6 secondary schools in Ngong Hub, Kajiado County, Kenya, between May and September 2025. The study evaluated the following two pilot aims: (1) the technical performance of shamiriAI’s ASR pipeline on a held-out test set of manually transcribed sessions, and (2) interrater reliability between shamiriAI-generated fidelity ratings and independent human supervisor ratings across 52 recorded sessions. These aims concern the technical properties of the system, not between-arm comparisons.

Lay providers (Fellows) were randomized to supervision arms at the start of the study. The sessions later recorded for validation were sampled independently at each site by lottery, without stratification by arm. Because assignment preceded session sampling, the validation set is imbalanced across arms (38 AI-augmented and 14 standard); the sampling mechanism is detailed under the Participants subheading.

The pilot was part of a larger 5-part implementation study examining optimization strategies for the Shamiri Model [41,42]. The complete protocol is available in Supplement A in Multimedia Appendix 1.

Lay providers (Shamiri Fellows) are Kenyan young adults aged 18‐24 years recruited and trained to deliver the Shamiri intervention (refer to the Procedures subheading). All 64 Shamiri Fellows assigned to Ngong Hub were eligible; sex and age were collected via self-report. The 47 Fellows who participated in the parallel A/B test were randomized at project start to AI-augmented (n=34) or standard (n=13) supervision.

Audio recording for fidelity validation was a separate procedure. At each school session, the Hub Coordinator wrote the names of all lay providers present on slips of paper and drew 3 without replacement for recording, with no stratification by arm. This produced 52 recorded sessions, distributed unevenly across arms (38 AI-augmented and 14 standard) and across providers, reflecting natural variation in attendance and recording opportunity rather than a prespecified arm-balanced design.

Lay providers led group sessions for students aged 12‐21 years participating in the routine Shamiri intervention ([21,24,28]; Multimedia Appendix 1: Supplement A). Students were assigned to groups using a ballot-box procedure.

Shamiri is a brief, school-based group intervention targeting adolescent depression and anxiety [21,24,28,29]. Its components derive from the science of strengths-based interventions (overlapping with “wise” interventions [43,44]): brief, simple techniques targeting specific psychological processes [45,46]—growth mindset [47,48], gratitude [49,50], and values affirmation [51,52]. Complete protocols are published elsewhere [24-26]; refer to Multimedia Appendix 1: Supplement A.

Supervisors recruited, trained, and supervised lay providers while ensuring fidelity, managed clinically elevated cases, and triaged emergencies to the Clinical Network of expert psychiatrists and psychologists [25-27]. All held at least a bachelor’s degree in clinical or counseling psychology and were registered with the Kenya Counselors and Psychologists Board. Each supervised approximately 10 lay providers (1:10). Recruitment used a written interview and 2 case interviews with the Shamiri Clinical Team [25,26]. Supervisors then completed approximately 3 months of training covering the intervention manual and session-by-session protocols; clinical supervision techniques, including structured feedback delivery, fidelity rating using the 6-domain instrument, and calibration exercises; group facilitation and peer-counseling skills; identification and triage of elevated cases through the Clinical Network; ethical conduct and professional boundaries; and emergency risk management via the validated Shamiri Risk Management Protocol [25-27]. The Shamiri Risk Management Protocol uses a 3-tier model (lay provider, caseworker or supervisor, and clinical expert) with standardized procedures for recognizing distress, initiating referrals, managing sensitive disclosures, and maintaining boundaries [26]. Supervisors were also trained in data entry and quality control using internal software [53].

Lay providers recorded group sessions using study-supplied digital voice recorders, which captured wide-angle audio in open or semiopen spaces (eg, classrooms, halls, and outdoor areas), producing variable background noise. Recording coverage varied across sessions due to device failures, environmental noise, and school-specific restrictions. We included recordings from all 52 delivered sessions for which a recording was available (100% of recorded sessions). In the AI-augmented arm, Hub Coordinators uploaded files to secure cloud storage after each session; in the standard arm, recordings were archived for potential later analysis but were not processed by shamiriAI during the trial.

In parallel with ASR, we extracted prosodic and conversational features using deterministic signal-processing techniques in librosa and related libraries. These features capture dimensions of group dynamics and facilitator behavior not reflected in transcribed text and were supplied as supplementary inputs to LLM inference [61-63].

Before transcripts reached the LLM, PII scrubbing used NuNerZero (numind/NuNerZero; NuMind), a zero-shot named entity recognition model based on the GLiNER architecture [70]. NuNerZero is a compact bidirectional token classifier trained on the NuNER (version 2.0) dataset, selected for its zero-shot capability (ie, requiring no labeled target-domain data) and support for arbitrary entity-label prompting at inference without fine-tuning. Three entity types (person, location, and organization) were targeted, and detected entities were replaced in place with typed placeholders ([REDACTED_PERSON], [REDACTED_LOCATION], and [REDACTED_ORGANIZATION]). Redaction was applied to ASR transcripts before any content reached LLM inference; audio and unredacted transcripts were retained on secure servers with restricted access. An English BERT (Bidirectional Encoder Representations from Transformers)–based NER model (dslim/bert-base-NER) was implemented as a fallback but not used in production. Formal precision and recall for the redaction pipeline were not computed in this pilot study.

Structured feedback reports were generated using Google Gemini 2.5 Pro (gemini-2.5-pro), selected after benchmarking against several contemporary LLMs, including GPT-based models, because of its superior performance on code-switched transcripts. The model was invoked once per session, receiving the diarized transcript and session-level audio features as structured JSON. Generation hyperparameters were fixed identically across all 52 sessions (Multimedia Appendix 1: Supplement C8): temperature of 1.0, top-p of 0.95, top-k of 40, maximum output of 65,536 tokens, and a dynamic thinking budget. The model returned a strictly typed JSON object validated against a Pydantic schema, which enforced integer scores on the 1‐7 scale for each rubric dimension and prevented continuous or out-of-range values. No postprocessing or regex extraction was applied; scores were obtained directly from the validated object.

Because no random seed was fixed, generation was stochastic, though score-level variance across calls was bounded by the discrete 1‐7 schema. To confirm this empirically, we ran a test-retest analysis on all 52 sessions by running the production prompt 3 times per session and computing the intraclass correlation coefficient (ICC [3,1]) across runs for each dimension. Test-retest ICC values ranged from 0.44 (Q6: Protocol Boundaries) to 0.74 (Q5: Clarity and Accessibility), with an overall composite ICC of 0.71, comparable to the human-human interrater reliability observed in this dataset (refer to Multimedia Appendix 1: Supplement C8.3 for full methods and results).

The structured system prompt (Multimedia Appendix 1: Supplement C9) provided (1) background on the Shamiri intervention, its session-by-session structure, and supervision goals; (2) the six fidelity domains (Required Contents, Specifics, Thoroughness, Clarity, Skill, and Purity), including the scoring rubric and behavioral anchors; (3) the redacted transcript and a session-level audio-features object; and (4) a request for structured feedback, including numerical ratings per domain, specific strengths, areas for improvement with transcript evidence, and suggested focus areas for the upcoming supervision session.

Prosodic features were passed as a raw JSON object with original numeric values (eg, spectral_centroid_mean: 2847.3; speech_gap_mean: 1.42); no bucketing, z-scoring, or label substitution was applied before prompt construction. The prompt provided an interpretation guide mapping each technical field to a plain-language description (eg, quick_exchange_ratio → “conversation flow and turn-taking patterns”; pitch_diversity → “emotional expressiveness and vocal variation”), instructing the model to use these descriptions rather than raw field names in its output. Per-segment prosodic features were embedded within each transcript turn object, keeping the acoustic signal colocated with the corresponding speech text in the model context. Whether prosodic features measurably improve fidelity scoring beyond transcript content alone was not isolated in this pilot and remains to be established.

The system prompt was developed iteratively across 3 versions using a development corpus of earlier Shamiri Hub recordings [21,28,29], distinct from and nonoverlapping with the 52 validation sessions. Refinements were driven by qualitative review of model outputs on this corpus and by progressively closer alignment with the Shamiri Intervention Protocol rubric. The final prompt was locked before assembly and scoring of the validation dataset; no validation session was reviewed, scored, or consulted during prompt development. The 52 sessions therefore constituted a clean held-out evaluation of the locked prompt, with no leakage between development and validation. Prompt-development artifacts were committed to version control together at project handoff rather than incrementally; therefore, file-level timestamps do not individually reflect chronological order, and the provenance account rests on author attestation rather than automated artifact evidence.

For each session, shamiriAI generated a PDF report containing session metadata (lay provider ID, school, session number, date, and duration), a quantitative summary of key prosodic indicators (eg, talk-time distribution, speech density, and quick-exchange ratio), numerical ratings per fidelity domain, and a narrative section structured around strengths, areas for improvement, and suggested focus areas. Reports were delivered by secure email or through a shared folder and reviewed during weekly supervision. Supervisors were encouraged to interpret AI-generated ratings and feedback alongside their own knowledge of lay providers and clinical judgment.

During the study, transcription and feedback generation were orchestrated via a Python pipeline, with human operators managing uploads and report distribution. Longer-term plans include integration into the Shamiri digital infrastructure (shamiriOS; Shamiri Institute) for automated ingestion, background processing, and in-platform feedback dashboards [53].

The study was approved by the Daystar University Institutional Scientific and Ethics Review Committee (DUISERC; approval DU-ISERC/10/06/2025/000015E; June 10, 2025) and licensed by the National Commission for Science, Technology and Innovation (NACOSTI; license NACOSTI/P/25/415077; issued February 11, 2025; valid through February 11, 2026). The trial was registered with the Pan African Clinical Trials Registry (PACTR202508900479778) on August 1, 2025. Audio recording at Ngong Hub began in May 2025, while routine program delivery operated under 2 existing approvals: the Kenyatta University Ethics Review Committee renewal (KUERC; application PKU/2627/E1752; renewed November 12, 2024; valid through November 12, 2025) for the broader “Testing Pathways to Scale for a Cost-Effective and Evidence-Based Mental Health Innovation” study, and the NACOSTI license (NACOSTI/P/25/415077) issued February 11, 2025 [41,42]. The A/B-test–specific DUISERC approval was received on June 10, 2025, and the first shamiriAI feedback reports were delivered to supervisors on June 20, 2025, after that approval was in place. PACTR registration was completed on August 1, 2025, after enrollment commenced; we acknowledge this as retrospective registration, which occurred because of in-house administrative transitions during a period of staff change. All data collection and AI-augmented supervision activities were conducted under existing DUISERC and KUERC approvals throughout. Written informed consent was obtained from all adult lay providers and adult student participants before involvement. For minors, written assent was obtained from the student, alongside parental or guardian consent secured through school administration, in accordance with DUISERC, KUERC, and NACOSTI protocols. Participants were informed that involvement was voluntary and that they could withdraw at any time without consequence. Lay providers were compensated at the standard Shamiri Institute rate of KES 1500 (approximately US $12) per 1-hour session delivered, consistent with their employment terms [25]. Student participants received no compensation. Audio recordings and unredacted transcripts were stored on secure servers with restricted access. Before any AI inference, personally identifiable information (PII), including names, school identifiers, and location references, was detected and masked using a named entity recognition pipeline and replaced with typed placeholders ([REDACTED_PERSON], [REDACTED_LOCATION], and [REDACTED_ORGANIZATION]). Only deidentified, redacted transcripts were used in LLM-based fidelity inference (refer to Stage 3 and Multimedia Appendix 1: Supplement C7). No participants are identified in any image or figure in this paper.

Of the 64 Shamiri Fellows assigned to Ngong Hub, 47 participated. Thirty-four were randomized to the AI-augmented supervision arm and 13 to standard supervision (Figure 1; Multimedia Appendix 1: Supplement D1). Most were female (36/47, 76.6%). Fellows ranged in age from 18 to 23 years (mean 19.83, SD 1.65). Across fellows, 52 recorded sessions were included in the fidelity analysis; most fellows contributed 1 or 2 recordings, and most rated sessions were from the AI-augmented arm (38/52, 73%). Seven supervisors participated; most were female (5/7, 71%), ranging in age from 23 to 30 years (mean 25.57, SD 2.23). Five (71%) supervisors were assigned to the AI-augmented arm and two (29%) to standard supervision. Sample characteristics are reported in Multimedia Appendix 1: Supplement D2.

The fine-tuned Whisper model (small checkpoint;~242 million parameters) was evaluated on a held-out test set of 10 sessions with manually prepared reference transcripts. The model achieved a CER of 0.19 and a WER of 0.34. Cosine semantic similarity (LaBSE-based) was 0.77, and Recall-Oriented Understudy for Gisting Evaluation–Longest Common Subsequence was 0.60. Segment-level Jaccard similarity averaged 0.72 (SD 0.26) and Jaro-Winkler similarity averaged 0.78 (SD 0.21). Full ASR metrics are provided in Table 2.

This pilot pursued 2 foundational aims for shamiriAI—an automated fidelity-monitoring system for lay-delivered group mental health interventions in multilingual, low-resource settings.

The first aim established that the fine-tuned Whisper pipeline produced transcripts of sufficient semantic quality for downstream fidelity evaluation in naturalistic, multispeaker, code-switched English-Kiswahili-Sheng audio. A WER of 0.34 looks high against English-only benchmarks, but zero-shot Whisper on Kiswahili commonly exceeds 0.50 on read speech [82,83]. After being trained on about 8 hours of domain data and operating on spontaneous group speech, our model represents a meaningful gain at minimal cost. The metrics most appropriate for this context—CER (0.19) and cosine semantic similarity (0.77)—indicate that session meaning is preserved even where individual tokens are misrecognized. The output is semantically coherent enough to support LLM-based inference.

The second aim produced a more complex picture. Across all 6 dimensions, shamiriAI rated sessions an average of 0.79 points lower than the human panel on average—a large aggregate effect (d=−1.16) that remained significant after Holm-Bonferroni correction. The model scored more critically than human supervisors. That headline obscures 3 distinct patterns with different implications for development.

The largest discrepancies were observed for Required Contents (d=−3.48) and Clarity (d= −1.56). The AI differed from the panel by more than one scale point in 94% of Required Contents sessions, and Clarity adjacent agreement was 40%. Both dimensions require holistic, integrative judgment about whether a session was complete and clear “in spirit,” not whether scripted elements appeared verbatim. Human supervisors credit spirit-compliant delivery—paraphrase, local idiom, and language mixing—as meeting the standard, whereas the AI appears to apply a more literal rubric, penalizing missing scripted phrases [84,85]. Required Contents may have a further diagnosable cause: ASR errors in group audio destroy the short transitional utterances that signal element completion, and required elements are often spread across turns and speakers rather than delivered in one block. These are architectural, not prompt-tuning, problems and point to specific version 2 fixes, including required-element checklists in the prompt, sequential per-element evaluation before an overall rating, behavioral anchors from calibrated annotations, and few-shot examples spanning acceptable delivery styles.

Two medium effects ran in opposite directions: the AI underrated Thoroughness (d=−0.99) and overrated Skill (d=+0.87). The opposing signs suggest that the model reads surface markers of facilitation—open questions, reflective language, validation, and restatement—as skill [85], but does not assess whether they were used at adequate depth, at the right moments, and with time for student reflection [86]. A provider producing dense facilitation language while rushing content earns high Skill but low Thoroughness; a supervisor seeing both together rates them more consistently. Prompt redesign should partly correct this by incorporating Thoroughness anchors distinguishing depth from technique frequency and Skill anchors weighting contextual appropriateness alongside behavioral presence, without new training data.

This is a positive finding. No significant bias was detected for Specifics (d=−0.34; P=.18) or Purity (d=+0.22; P=.22), and adjacent agreement reached 78.8% and 73.1%, approaching the human–human AC2 ceiling of 0.42‐0.60. Both are detection tasks—whether prescribed content is present and prohibited content was absent—and on these shamiriAI already performs at a level that could support supervisory decisions. This matters because Specifics and Purity are the dimensions most tied to protocol adherence and clinical safety, the quality floor that supervision exists to enforce. An AI system that flags off-protocol sessions and confirms coverage already gives supervisors actionable signals on what matters most for preventing drift [86,87].

The reliability indices must be interpreted relative to 2 intrinsic properties of the human reference standard. First, human composite ratings clustered near the ceiling (mean 5.78‐6.14, SD 0.58‐0.74), restricting variance and mechanically suppressing ICCs regardless of true rank alignment. This explains why Specifics and Purity were classified as “poor” by ICC benchmarks [81,88] yet exceeded 70% adjacent agreement. Gwet AC2, with prevalence- and bias-adjusted chance correction, partly corrects this and confirms the dimension ordering. Second, human-human AC2 ranged from 0.42 to 0.60, representing the realistic upper bound any model could achieve against this particular composite [89]. AI-human AC2 on Specifics, Skill, and Purity approaches that ceiling rather than falling short of an unattainable maximum.

The 52-session sample is small for stable AC2 estimation, and CIs are wide (Tables 3 and 4). Accordingly, these estimates require replication in a planned larger cohort (target N=400). The direction is more secure: the dimension ordering—highest for Specifics, Skill, and Purity; lowest for Required Contents; and intermediate for Thoroughness and Clarity—held across the ICC analysis, both AC2 formulations, and the sex, age, and arm subgroups, suggesting that it is not an artifact of the reference treatment or demographic or arm-level confounding. These subgroup analyses were exploratory and underpowered, excluding only large effects.

Possibly comparable existing systems achieve higher reliability but under far more favorable conditions. Lyssn (Lyssn.io), a commercial platform trained on about 2500 labeled sessions of individual CBT by professional therapists in English, reaches 100% of human reliability on the overall quality score and exceeds 80% on 10 of 11 items [36]. A recent LLM system scored patient engagement across 1131 individual CBT sessions in German with strong reliability and outcome-linked validity [40]. shamiriAI currently achieves lower reliability, but the conditions differ fundamentally: individual vs group, professional vs lay providers, 1 language vs 3 including code-switched Sheng, large labeled corpora vs 8 hours, controlled vs open-environment audio. The relevant benchmark is the gap: on Specifics and Purity, shamiriAI already approaches the range of systems built for easier conditions, and its Required Contents and Clarity limitations are interpretable and addressable rather than fundamental. Procedural prompting has been shown to substantially improve LLM assessment reliability [90], directly applicable to the priorities here.

This pilot has several limitations that constrain inference and set priorities for planned follow-up studies (target N=400) and future development.

First, the 52-sessions sample is small for stable agreement estimation in a ceiling-affected distribution; AC2 CIs are wide, and a larger sample is needed.

Second, the human reference showed only moderate interrater reliability (AC2 0.42‐0.60) and a restricted range (mean 5.78‐6.14, SD 0.58‐0.74), capping achievable agreement. Future iterations should incorporate structured rater calibration training and behavioral-anchored formats.

Third, the validation set was sampled without stratification by arm, producing a 38:14 imbalance. Per-arm comparisons detected no systematic bias, but validation with arm-balanced samples is needed.

Fourth, prompt-development provenance rests on author attestation rather than incremental timestamps, because artifacts were committed together at handoff; all 3 iterations are archived on Open Science Framework. Future development should adopt incremental version control.

Fifth, the PII redaction pipeline was not formally evaluated for precision and recall. Person-name redaction is qualitatively acceptable, but location and organization redaction is plausibly weaker for Sheng and Kiswahili. Accordingly, we treat NER as a supplementary layer behind secure storage and consent.

Sixth, audio recordings spanned 3 native sampling rates without resampling, so spectral features, MFCCs, and pitch outputs are not strictly comparable across sessions. Cross-rate normalization should be added in a v2.

Seventh, the zero-shot ASR baseline is drawn from published benchmarks [54,57,58] rather than a within-corpus run, so the fine-tuning gain cannot be precisely quantified; ASR metrics were also computed on a separate held-out test set, not the 52 fidelity validation sessions.

Eighth, diarization DER was not measured on the Shamiri corpus; we report published benchmark DER for pyannote [59,60] as a proxy, and actual DER in adolescent group sessions may be worse.

Ninth, a prosodic-feature ablation was deferred to the larger cohort, where it can be adequately powered rather than interpreted post hoc.

Tenth, all sessions came from a single hub in Kajiado County; generalizability to other hubs, regions, languages of code-switching, or non-Shamiri interventions must be tested in multisite replication.

Eleventh, the trial was registered with PACTR after enrollment commenced; we acknowledge this departure from prospective registration and note that all activities fell within the DUISERC and KUERC approval period.

Twelfth, raters were calibrated before coding but did not recalibrate during the rating window and could not see one another’s scores, so within-trial drift cannot be fully excluded.

Thirteenth, Session 4 was underrepresented (6/52, 11.5%), so its reliability should be reestimated in the larger cohort.

Fourteenth, scores come from a generative model sampled at temperature 1.0, so identical inputs can vary slightly across runs; test-retest agreement across 3 runs was high, and deterministic scoring would require temperature 0 or aggregation across runs.

Finally, this study evaluates technical performance only. Whether AI-augmented supervision improves provider skill, fidelity at scale, or student outcomes remains for the planned cluster-randomized noninferiority trial.

The goal of shamiriAI is not fidelity monitoring for its own sake but a quality-assurance infrastructure that can support lay-provider supervision at population scale—the precondition for task-shifted care to reach the millions of young people who receive no help [18,33,35,91,92]. This pilot establishes the 2 foundations required: a transcription pipeline that works in multilingual, code-switched, naturalistic group speech, and a fidelity-rating system whose failures are diagnosable rather than diffuse.

The immediate priority is shamiriAI (version 2)—improved ASR through expanded fine-tuning, prompt redesign for Required Contents, Clarity, and Thoroughness, and integration into shamiriOS [53] to deliver automated fidelity reports for every session at every active site, replacing the 10%‐15% of sessions that now receive any human review [26,93]. Once cross-dimension reliability reaches an adequate threshold, the next study is a fully powered cluster-randomized noninferiority trial comparing AI-augmented with standard supervision on youth depression and anxiety outcomes, provider skill, and cost-effectiveness. Beyond supervision, the session-level data shamiriAI generates can support causal mediation analyses, precision matching of students to providers, and identification of active ingredients [94]—a learning-system capacity that distinguishes it from a stand-alone monitoring tool and positions it as the backbone of an AI-native stepped-care model.

The work’s importance rests on the supervision bottleneck. Kenya has about 2 specialized mental health workers per 100,000 people; sub-Saharan Africa has 1.4‐3.8 [8,9]. Task-shifting is the primary response to this gap and its effectiveness is well established [18,30,91], but supervision is the binding constraint on quality and scale [18,95,96]. AI fidelity monitoring does not replace supervisors; it gives them the reach to supervise hundreds of providers across thousands of sessions. This pilot is the first evidence that such a system is technically buildable where it is most needed.

The pilot specifies a concrete version 2 agenda: prompt redesign for Required Contents and Clarity with session-specific checklists and behavioral anchors, sequential per-element evaluation, Thoroughness and Skill anchors separating technique frequency from depth and timing, cross-rate feature normalization, and within-corpus evaluation of the zero-shot ASR baseline, diarization error, PII recall, and the prosodic ablation, all on the larger cohort. In parallel, future development should adopt incremental prompt version control with a dated changelog, prespecify arm-balanced sampling, audit PII redaction recall, and add calibration training for human raters. Beyond version 2, multihub replication and extension to non-Shamiri interventions and other low- and middle-income country contexts should inform version 3.

Within a 52-session pilot, this study establishes 2 foundational results for shamiriAI. The fine-tuned multilingual Whisper pipeline achieved CER of 0.19 and cosine semantic similarity of 0.77 on naturalistic, multispeaker, code-switched English-Kiswahili-Sheng sessions, sufficient to support downstream LLM-based fidelity inference. The fidelity-rating system produced a dimension-dependent reliability profile: low ICCs against the continuous human composite (−0.06 to 0.20), attributable to restriction of range, with AC2 sensitivity analyses corroborating substantial agreement on Specifics, Skill, and Purity (0.69‐0.76, approaching the human-human ceiling of 0.42‐0.60) and systematic underrating on Required Contents and Clarity. The results characterize current technical performance only; whether AI-augmented supervision improves provider skill, fidelity at scale, or student outcomes will be tested in the planned cluster-randomized noninferiority trial.