This study was reviewed by the Korea University Anam Hospital Institutional Review Board and was approved for an ethics exemption (2025AN0508). All users of the Maeumpyeonuijeom app provided informed consent for the research use of their anonymized data at the time of registration.

After fine-tuning, we optimized 2 inference parameters—temperature and top_p—to improve response quality [22]. Manual optimization across multiple parameter combinations was impractical; therefore, we used GPT-5 as an automated judge to evaluate response quality across a grid of parameter settings [23]. The LLM judge evaluated generated responses on the same 4 dimensions used in human evaluation: individualization, supportiveness, cultural appropriateness, and safety.

A systematic grid search was conducted for each model over decoding parameters known to influence generation diversity and coherence. Temperature was varied from 0.4 to 1.2 in increments of 0.2, and top_p from 0.3 to 1.0 in increments of 0.1 [22]. For each parameter combination, responses were evaluated using an LLM-as-a-judge framework across the same 4 clinical dimensions used in human evaluation: individualization, supportiveness, cultural appropriateness, and safety [23]. The optimal configuration was defined a priori as the parameter setting with the highest mean score across the 4 dimensions.

On the basis of this search, LG EXAONE 3.5 and SKT A.X 4.0 Light were both optimized at a temperature of 0.8 and top_p of 0.8. For Meta Llama 3.1, the temperature was set to 0.8, but top_p was reduced to 0.5; higher top_P values were found to increase the occurrence of English tokens in generated responses, attributable to this model’s comparatively limited Korean-language pretraining, and constraining top_p to 0.5 effectively suppressed this linguistic drift while yielding the highest mean LLM-as-a-judge evaluation score among the tested configurations. Full details of the LLM-as-a-judge evaluation framework, including the grid search procedure, final selected parameters, and evaluation system prompt, are provided in Multimedia Appendix 1.

For the SLMs, generation parameters were tuned using an LLM-as-a-judge framework because of the limited size of the domain-specific dataset. Candidate temperature and top_p settings were evaluated, and the configuration yielding the most appropriate and stable responses was selected for inference. In contrast, GPT-4o was used as a general-purpose benchmark without domain-specific parameter tuning. The default OpenAI API generation settings were therefore used: temperature=1.0, top_P =1.0, frequency penalty=0, and presence penalty=0, with one response generated per prompt (n=1). No explicit maximum output-token limit or custom stopping criterion was specified.

We calculated intraclass correlation coefficients (ICCs) using a 2-way random-effects model for average measures (ICC{2,k}) to assess agreement among psychiatrist evaluators for each model and criterion combination. ICC values were interpreted as poor (<0.40), fair (0.40-0.59), good (0.60-0.74), and excellent (≥0.75) [13].

Wilcoxon signed-rank tests with Holm correction for multiple comparisons were used to compare each SLM against GPT-4o across evaluation criteria. Nonsignificant results (P>.05) were interpreted as no statistically significant difference detected.

Spearman correlation coefficients were calculated between LLM judge scores and mean psychiatrist scores to validate the LLM-as-a-judge methodology.

Table 1 presents the demographic characteristics of users in the training and test datasets. The training set participants had a mean age of 33.78 (SD 11.75) years, with 70.3% (267/380) female representation. The mean Patient Health Questionnaire-9 (PHQ-9) score was 12.03 (SD 6.27), indicating moderate depression severity on average. The test set showed similar characteristics, with a mean age of 32.36 (SD 10.17) years and 92.9% (26/28) female representation.

Table 2 presents the mean scores and SDs for each model across the 4 evaluation criteria. The overall performance trends are visualized in Figure 1, which illustrates the comparative mean scores of the 4 models. As shown in Figure 1, the SKT A.X 4.0 Light demonstrates a competitive performance trajectory, closely mirroring GPT-4o in the supportiveness and cultural appropriateness dimensions.

To provide a more granular view of performance beyond mean scores, Figure 2 illustrates the detailed distribution of ratings assigned by the 10 psychiatrists (N=500 ratings per criterion). As shown in Figure 2, the SKT A.X 4.0 Light achieved a competitive distribution of favorable ratings (scores of 4 or 5) in supportiveness and cultural appropriateness, comparable to GPT-4o. In contrast, for the safety criterion, GPT-4o maintained a dominant proportion of high scores (86% rated 4 or 5), while SLMs showed a wider spread with more ratings in the neutral (score 3) range.

Table 3 presents the results of Wilcoxon signed-rank tests comparing each SLM to GPT-4o.

No statistically significant difference from GPT-4o was detected for the SKT AX 4.0 Light in 2 dimensions: supportiveness (P=.26) and cultural appropriateness (P=.58). This indicates that, under our evaluation setting, the 7-billion-parameter SKT AX 4.0 Light showed comparable performance to GPT-4o in these dimensions, though this does not constitute formal statistical equivalence. All other comparisons showed significant differences favoring GPT-4o (P<.001), particularly in safety.

To further validate this equivalence, we conducted a pairwise head-to-head comparison (Figure 3). As illustrated, the SKT A.X 4.0 Light achieved a noninferiority rate (win + tie) of 84.2% in cultural appropriateness and 70.2% in supportiveness against GPT-4o. This indicates that in most counseling interactions, the locally fine-tuned SLM provided responses that were competitive with those of the general-purpose proprietary model.

Table 4 presents ICC values indicating the level of agreement among psychiatrist evaluators. A notable pattern emerged: GPT-4o consistently showed the lowest interrater reliability across all criteria, with particularly poor agreement for cultural appropriateness (ICC=0.211). In contrast, the locally fine-tuned models, especially SKT A.X 4.0 Light, achieved higher agreement levels (ICC=0.575 for cultural appropriateness), suggesting that psychiatrists more consistently recognized the cultural appropriateness of locally adapted responses.

Table 5 compares the LLM judge (GPT-5) evaluation results with human evaluation results. Figure 4 visualizes the rating distribution generated by the LLM judge, revealing distinct evaluation patterns compared with human psychiatrists.

As evident in Figure 4, the LLM judge exhibited a significant ceiling effect in the safety criterion, assigning perfect scores (5) to a vast majority of responses across all models. This contrasts sharply with the human evaluation in Figure 2, where safety ratings were more varied. This ceiling effect indicates that the LLM judge was unable to differentiate safety performance across models, though the human expert evaluation successfully identified the significant safety advantage of GPT-4o over all SLMs.

This study demonstrated that an SLM with approximately 7 billion parameters, when appropriately fine-tuned on culturally specific data, could achieve performance comparable to GPT-4o in key dimensions of mental health counseling quality. The SKT A.X 4.0 Light achieved no significant difference from GPT-4o in supportiveness (P=.26) and cultural appropriateness (P=.58), despite substantially smaller model size.

This finding may have implications for improving the accessibility and local deployability of AI-assisted mental health services. The strong performance achieved with less than 1% of the parameter count suggests that, in domain-specific and culturally grounded applications such as mental health counseling, data quality and cultural alignment can substantially influence model performance alongside model scale. Notably, the SKT A.X 4.0 Light ranked first among all evaluated models—including GPT-4o—in cultural appropriateness. This result underscores a fundamental limitation of globally trained large models: their tendency toward Western-centric biases can result in responses that feel foreign or inappropriate to users from different cultural backgrounds [24]. The strong performance of SKT A.X 4.0 Light in this dimension supports the potential value of culturally specific fine-tuning for mental health applications.

Our findings have several important implications for clinical deployment of AI-assisted mental health tools.

First, regarding privacy and accessibility, the ability to achieve GPT-4o-level performance with a locally deployable model addresses critical data privacy concerns. Health care institutions may benefit from locally deployable mental health AI systems because such systems can reduce the need to transmit sensitive patient data to external servers, a concern that is particularly salient under health information privacy frameworks such as the Health Insurance Portability and Accountability Act (HIPAA) [25]. In this study, our results suggest that competitive performance may be achievable using smaller models deployed in closed-network environments.

Second, regarding cultural adaptation, the competitive cultural appropriateness ratings for locally tuned models, relative to a strong general-purpose benchmark, provide an empirical foundation for the argument that mental health AI must be culturally adapted rather than simply translated. This aligns with established principles in cross-cultural psychology emphasizing the importance of cultural context in therapeutic communication [26].

Third, regarding cost efficiency, the dramatic reduction in model size translates to substantially lower computational requirements and operational costs, potentially improving access to AI-assisted mental health support in resource-limited settings.

This safety gap supports the need for human expert oversight when AI systems are used in mental health contexts, particularly given the risks associated with high-stakes or crisis-related interactions [27]. This gap is clinically significant and indicates that current SLMs are not ready for autonomous deployment in mental health settings. Rather, it highlights the necessity of a modular safety architecture for SLM-based systems. Unlike monolithic cloud-based models where safety mechanisms are opaque and immutable, locally deployed SLMs enable the integration of customizable, transparent safety guardrails—such as keyword-based filters for self-harm content, secondary classifier models for risk detection, or rule-based response modification [28]. Furthermore, this safety gap reinforces the principle that no AI system should operate autonomously in mental health contexts; human expert review remains essential regardless of model capability [29]. The combination of culturally competent SLMs with explicit safety layers and human oversight may ultimately provide a more controllable and auditable safety framework than reliance on proprietary models alone. Importantly, the lower safety scores of SLMs may partially reflect a trade-off with cultural authenticity—GPT-4o’s higher safety scores could be associated with more conservative, generic responses that sacrifice emotional engagement for caution, as suggested by its lower cultural appropriateness ratings.

A particularly intriguing finding was the consistently lower interrater reliability for GPT-4o compared with locally tuned models. Psychiatrists showed markedly poor agreement (ICC=0.211) when evaluating GPT-4o’s cultural appropriateness, compared with moderate agreement for SKT A.X 4.0 Light (ICC=0.575). This pattern suggests that GPT-4o’s responses may lack the consistency in cultural adaptation that allows experts to reach consensus on their appropriateness.

This inconsistency may reflect GPT-4o’s tendency to produce responses that occasionally align well with Korean cultural norms while at other times exhibiting distinctly Western communication patterns. Such variability would naturally lead to divergent expert opinions depending on which aspect of the response individual evaluators prioritize.

Our 2-stage hybrid validation framework proved effective for efficient model development. The LLM judge correctly identified relative performance rankings among models and demonstrated good correlation with human evaluation for supportiveness. However, important limitations emerged.

The LLM judge exhibited a significant ceiling effect in the safety dimension, assigning perfect scores (5) to a vast majority of responses across all models and thus failing to differentiate safety performance between GPT-4o and the SLMs. The notably higher safety scores observed for GPT-4o in human expert evaluation may reflect the extensive safety alignment procedures commonly incorporated into GPT-4-series models, including reinforcement learning from human feedback and adversarial red-teaming [30]. However, this interpretation remains inferential, and we cannot exclude the possibility that additional unmeasured factors contributed to the observed safety differences. This insensitivity in automated safety scoring is particularly consequential for mental health applications where safety discrimination is critical. We therefore recommend that LLM-as-a-judge methodologies be used for preliminary screening and hyperparameter optimization, but final validation for clinical applications must include human expert evaluation, particularly for safety assessment [23].

Our success in developing a competitive Korean mental health AI has implications for the broader goal of providing culturally appropriate mental health support globally. However, several conditions enabled this achievement.

First, the availability of high-quality Korean base models (SKT A.X 4.0 Light and LG EXAONE 3.5) was essential. These models already possess deep Korean-language understanding that our fine-tuning could build upon. Second, our dataset comprised professionally reviewed counseling responses, ensuring high-quality examples for the model to learn from. Third, the involvement of psychiatrists and licensed clinical psychologists in data preparation and evaluation ensured clinical validity.

These requirements highlight a potential equity concern: underresourced languages lacking high-quality base models may not be able to replicate our approach through fine-tuning alone. Additional pretraining on language-specific corpora may be necessary, representing a substantially higher barrier [31]. Addressing this gap should be a priority for global digital mental health equity.

Several limitations should be considered when interpreting our findings.

First, the dataset and evaluation scale present inherent constraints. The training corpus of approximately 1500 samples is modest relative to large-scale language model benchmarks, and while our aggressive fine-tuning strategy was designed to maximize learning from limited but high-quality data, larger datasets may enable further performance gains or enhance generalizability. Similarly, our human evaluation involved 50 counseling scenarios assessed by 10 psychiatrists; a larger and more diverse evaluation set would strengthen confidence in the generalizability of findings across the broad spectrum of mental health concerns encountered in clinical practice. Although the evaluation scenarios were temporally separated from the training data, this held-out evaluation provides only indirect evidence against memorization and does not constitute a formal test of memorization or a validated assessment of distributional generalization; verbatim memorization of training examples cannot be fully excluded. It should be noted that the dataset scale was additionally constrained by the availability of a time-limited GPU resource program, which has since concluded.

Second, the test set demographics showed high female representation (26/28, 92.9%) and predominantly moderate depression severity (mean PHQ-9 score 10.86). This profile reflects the user base of the source application but limits generalizability to male users, individuals with severe depression, or those with other conditions such as anxiety disorders, bipolar disorder, or psychotic disorders. The training data exhibited similar demographic patterns, reinforced through aggressive fine-tuning. The high female representation may have introduced gender-specific linguistic patterns—including vocabulary, emotional framing, and empathic tone—into the model’s learned communication style, potentially reducing performance for male users whose expression of emotional distress may differ systematically. Future studies should include active recruitment of male participants and individuals across diverse age groups and psychiatric diagnoses.

Third, our evaluation was conducted in a controlled diary-response format, representing a specific subset of mental health communication contexts. Performance in synchronous chat, crisis intervention, or live therapeutic conversation may differ substantially. Furthermore, models were evaluated on structured prompts; real-world inputs including grammatically incorrect text, colloquial expressions, or ambiguous emotional content may yield variable performance. The long-term effects of AI-assisted counseling on clinical outcomes—including changes in depression symptoms, user satisfaction, and sustained engagement—were not assessed, and the clinical significance of the observed statistical equivalence requires validation through longitudinal outcome studies.

Fourth, this study did not use standard automated natural language processing evaluation metrics such as BLEU (Bilingual Evaluation Understudy), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), BERTScore (Bidirectional Encoder Representations from Transformers), or perplexity. This was an intentional methodological decision: word-overlap metrics such as BLEU and ROUGE have been empirically shown to exhibit limited correlation with human judgments of dialogue response quality [19], embedding-based metrics such as BERTScore do not adequately capture the empathic creativity essential to psychiatric counseling, and fluency-based metrics such as perplexity do not reflect clinical and empathic dimensions central to therapeutic effectiveness. Additionally, the use of a single model (GPT-5) as an automated judge may introduce model-specific biases; multimodel LLM-as-a-judge frameworks could improve robustness, though the clinical impact of this limitation is minimized by the reliance on human expert evaluation for all primary findings. Future work incorporating complementary computational metrics and multimodel automated evaluation alongside expert evaluation would provide a more comprehensive characterization of model outputs.

Fifth, formal ablation studies comparing our aggressive fine-tuning configuration (LoRA rank r=256, 20 epochs) with more conventional configurations (eg, lower LoRA ranks and fewer training epochs) were not performed due to the constraints of the time-limited GPU resource program. This represents a genuine limitation, as we cannot definitively confirm whether the specific hyperparameter choices were strictly necessary, and we have noted the absence of a formal ablation study accordingly.

Sixth, 2 practical limitations constrain the broader applicability of this work. Our approach depended critically on the availability of high-quality Korean base models; languages without comparable foundation models may face substantial barriers to replicating this methodology, representing a potential equity concern for global digital mental health. Additionally, while reduced operational cost is cited as a primary motivation for SLM adoption, this study did not benchmark inference latency, throughput, or comparative compute costs between local SLM deployment and GPT-4o API calls. Rigorous cost-benefit analysis in a real-world clinical infrastructure environment remains an important direction for future research.

Seventh, because GPT-4o was not equivalently domain-adapted, this comparison should be interpreted as benchmarking against a general-purpose model rather than as demonstrating the superiority of the SLM or the fine-tuning strategy.

This study demonstrated that SLMs, when strategically fine-tuned on culturally specific data, could achieve performance comparable to state-of-the-art LLMs in mental health counseling applications. The SKT A.X 4.0 Light (7 billion parameters) showed no statistically significant difference from GPT-4o, a frontier-scale proprietary language model, in supportiveness and cultural appropriateness as evaluated by psychiatrists, while using less than 1% of the parameters.

These findings support the feasibility of deploying effective, privacy-preserving, culturally appropriate mental health AI on local infrastructure. However, the persistent safety gap between SLMs and larger models necessitates careful implementation with appropriate safeguards, including modular safety guardrails and mandatory human oversight. Furthermore, the success of this approach depends on the availability of high-quality language-specific base models, highlighting an important consideration for global digital mental health equity.

Our 2-stage hybrid validation framework combining LLM-as-a-judge screening with expert clinical evaluation offered a practical methodology for developing and validating domain-specific AI applications, though the inability of automated LLM judges to discriminate safety performance across models reinforced the irreplaceable role of human expert oversight in clinical AI development.