This study adopts a structured approach to the continuous pretraining and evaluation of 2 NorDeClin-BERT, new clinical BERT-based models developed for predicting ICD-10 codes from Norwegian clinical notes, specifically focusing on the gastroenterology domain. This section covers ethical considerations related to data use, the process of data collection and preparation, selection of model architecture and continuous pretraining, fine-tuning, evaluation, and interpretability analysis.

This research was approved by the Norwegian Regional Committees for Medical and Health Research Ethics North, decision number 260972. This study is based on a retrospective analysis of deidentified clinical text. The ethics committee granted a waiver of informed consent in accordance with Norwegian regulations for secondary use of health data in research. All data used in this study were fully deidentified prior to analysis. No personal identifiers were included in the dataset. Access to the data was restricted to authorized personnel, and all analyses were conducted in secure computing environments. No compensation was provided to individuals, as the study did not involve direct participation and was conducted on retrospective clinical data. The manuscript does not contain any images or materials in which individual participants or users can be identified. All data used in this study were fully deidentified prior to analysis. No personal identifiers were included in the dataset. Access to the data was restricted to authorized personnel, and all analyses were conducted in secure computing environments.

All data used in this study were fully deidentified prior to analysis. No personal identifiers were included in the dataset. Access to the data was restricted to authorized personnel, and all analyses were conducted in secure computing environments.

This study presents 2 versions of NorDeClin-BERT, both developed through continuous pretraining on Norwegian clinical text. The first, NorDeClin-BERT-base, is based on the NorBERT3-base architecture, consisting of 12 transformer layers, 12 self-attention heads per layer, and a hidden size of 768 dimensions, resulting in 123 million parameters. The second, NorDeClin-BERT-large, builds upon NorBERT3-large, which features 24 layers, 16 attention heads per layer, a hidden size of 1024 dimensions, and approximately 340 million parameters. These architectures were selected for their proven effectiveness in capturing contextual information and learning rich language representations.

While both NorDeClin-BERT models retain the architecture of their respective backbone models, they differ in the domain-specific knowledge acquired through further pretraining. NorBERT3-base and NorBERT3-large were originally pretrained on general-domain Norwegian text, whereas we further trained both models on deidentified and pseudonymized Norwegian clinical text to create NorDeClin-BERT-base and NorDeClin-BERT-large, respectively. This continuous pretraining process enhanced the models’ ability to understand and represent medical language more effectively, making them well-suited for downstream clinical tasks such as ICD-10 code prediction.

To further pretrain the NorDeClin-BERT models on our specialized clinical text data, we used a well-structured training pipeline built upon the Hugging Face Transformers library [32]. The pretraining process was carried out on a Republic of Gamers server running Debian Linux, initially equipped with a single ASUS GeForce RTX 3090 GPU and later expanded to support dual GPUs for training larger configurations. The system has 64 GB of RAM (2×32GB 3200 MHz DDR4), and an 8 TB Gen4×4 M.2 NVMe SSD. The server storage was encrypted and located in a secure server room, accessible only to researchers who were specially authorized to work with the data and had signed confidentiality agreements. The server was not connected to the internet to ensure data security and remained offline throughout the project.

NorDeClin-BERT was continuously pretrained using the masked language modeling (MLM) objective. In MLM, a portion of the input tokens is randomly masked, and the model is trained to predict the original tokens based on the surrounding context. This approach allows the model to learn robust representations of words and their relationships. Following the findings from the RoBERTa paper [31], which indicated that the next-sentence prediction task was unnecessary, we opted to focus exclusively on MLM during the pretraining of both versions of NorDeClin-BERT.

The tokenized data parts were loaded and concatenated to form a complete training dataset. The dataset was designed to be dynamically masked during training, where tokens were randomly masked at a probability of 15% to train the model on the MLM objective. Training parameters were carefully configured to optimize the model’s performance, closely following the RoBERTa paper [31]. Both NorDeClin-BERT-base and NorDeClin-BERT-large were pretrained for 40 epochs with a learning rate of 0.0001.

For NorDeClin-BERT-base, the batch size was configured for 8 sequences per device, with gradient accumulation steps set to 16, effectively simulating a larger batch size of 128 sequences. For NorDeClin-BERT-large, the configuration was adapted for dual-GPU training, using a batch size of 16 and accumulation steps of 2 per device, yielding an effective batch size of 64. While not identical, these settings were selected to maintain stable training dynamics under different hardware constraints. Additionally, the training process included a warmup phase (10,000 steps for base, 5000 for large), weight decay of 0.01, and no gradient clipping. The Adam optimizer was used with custom ε of 0.000001, β₁ of 0.9, and β₂ of 0.999. During the training process, checkpoints were saved periodically, with retention limits in place to manage disk space efficiently. The training could be resumed from a specific checkpoint if needed.

After continuous pretraining, both NorDeClin-BERT-base and NorDeClin-BERT-large were fine-tuned for ICD-10 code prediction using 87,938 discharge summaries with K-codes. The dataset was partitioned into training (70,350/87,938, 80%), validation (8794/87,938, 10%), and testing (8794/87,938, 10%) sets. The fine-tuning process began with data preparation, where the discharge summaries and their corresponding ICD-10 codes were loaded from a CSV file using the Hugging Face datasets library. Each summary’s codes were split into a list format for further processing. Figure 1 illustrates this workflow, showing how the NorDeClin-BERT models were pretrained on Norwegian clinical texts and subsequently fine-tuned on the Norwegian ICD-10 coding task to create the final classification models.

A custom preprocessing function was implemented to tokenize the text and prepare the labels. This function tokenized each discharge summary using the model’s tokenizer with a maximum sequence length of 512 tokens. Labels were encoded using a multihot encoding scheme, where each unique ICD-10 code was represented as a binary vector. The NorDeClin-BERT models were then loaded with a classification head adapted for multilabel classification, with the number of output labels set to match the total number of unique ICD-10 codes in the dataset.

The training setup used a custom MultilabelTrainer class, extending the HuggingFace Trainer class for multilabel classification. The trainer used a binary cross-entropy loss function BCEWithLogitsLoss and was configured with specific hyperparameters: 40 epochs, a learning rate of 2e-5, and an early stopping patience of 1 epoch. To effectively manage memory constraints and increase the batch size, the training used a batch size of 4 with 16 gradient accumulation steps, resulting in a batch size of 64.

During the fine-tuning process, the model was trained on the prepared dataset, with evaluation performed on the validation set after each epoch. Early stopping was applied to prevent overfitting, and the best model was saved based on validation performance. The training process used a constant learning rate scheduler.

After training, the models were evaluated on the held-out test set using custom metric functions to compute accuracy, precision, recall, and F₁-score for multilabel classification. A threshold of 0.5 was applied to the model’s output probabilities to determine the final predictions.

To provide insights into the decision-making processes of the NorDeClin-BERT models, an attention-based interpretability analysis was conducted. This involved generating a synthetic clinical text using ChatGPT, processing the text through both NorDeClin-BERT-base-NorICD and NorDeClin-BERT-large-NorICD models, extracting attention weights, aggregating and normalizing attention weights across all layers and heads, and visualizing attention distribution across input tokens during ICD-10 code prediction.

This methodology allows for a comprehensive evaluation of the NorDeClin-BERT models’ performance, their comparative advantages over other BERT variants, and insights into their internal decision-making processes, all crucial for assessing their potential in automating ICD-10 coding in Norwegian health care settings.

The evaluation of the NorDeClin-BERT models and their comparison with other BERT-based models across 4 critical metrics (accuracy, precision, recall, and F₁-score) yielded meaningful insights into their performance on ICD-10 code classification tasks. The analysis was conducted for 2 distinct scenarios: classification performance for all codes and the top 80% most frequently used codes. Performance was further categorized into multilabel and top-5 accuracy.

Table 3 presents the accuracy scores across all evaluated models under 4 evaluation settings. NorDeClin-BERT-large-NorICD achieved the highest accuracy across all scenarios, including multilabel (0.47) and top-5 (0.82) classification of the full ICD-10 code set, as well as multilabel (0.56) and top-5 (0.88) classification of the top 80% most-used codes. It outperformed all other models, including the larger general-domain NorBERT3-large-NorICD, with the largest margin observed in the all codes multilabel setting (0.47 vs 0.42).

Among base-sized models, NorDeClin-BERT-base-NorICD also showed strong performance, surpassing SweDeClin-BERT-NorICD, ScandiBERT-NorICD, and SweDeClin-BERT-SweICD in all settings. Notably, it matched or exceeded the performance of the larger NorBERT3-large-NorICD in 3 out of 4 scenarios, highlighting the impact of clinical domain adaptation even in smaller models.

Table 4 presents precision scores across all models and evaluation scenarios. NorDeClin-BERT-large-NorICD achieved the highest precision in 3 out of 4 settings, including all codes multilabel (0.66), top-5 (0.82), and top 80% most-used codes top-5 (0.90). It performed comparably to NorBERT3-large-NorICD in the remaining setting, where NorBERT3-large-NorICD achieved a higher top 80% most-used codes multilabel precision (0.73 vs 0.72).

Among base-sized models, NorDeClin-BERT-base-NorICD outperformed all other base models in every scenario, with precision scores of 0.65 (all codes multilabel), 0.80 (top-5), 0.71 (top 80% most-used codes multilabel), and 0.89 (top 80% most-used codes top-5). This performance closely approaches that of the large models, further reinforcing the strength of domain-specific pretraining even with smaller architectures.

Table 5 reports the recall scores across all models and evaluation scenarios. NorDeClin-BERT-large-NorICD consistently achieved the highest recall across all 4 evaluation settings, with scores of 0.48 (all codes multilabel), 0.82 (all codes top-5), 0.54 (top 80% most-used codes multilabel), and 0.88 (top 80% most-used codes top-5). The largest improvement was observed in the multilabel settings, where it outperformed the general-domain NorBERT3-large-NorICD by 5% points in all codes (0.48 vs 0.43) and 4 points in the top 80% codes (0.54 vs 0.50), underscoring the advantage of domain-specific pretraining at scale.

Among the base-sized models, NorDeClin-BERT-base-NorICD also performed strongly, achieving 0.45 recall in all codes multilabel, 0.81 in top-5, 0.51 in top 80% most-used codes multilabel, and 0.87 in top 80% most-used codes top-5. It outperformed all general-domain baselines (ScandiBERT and NorBERT3-base), as well as the domain-specific Swedish models (SweDeClin-BERT-SweICD and SweDeClin-BERT-NorICD). Its recall closely matched or exceeded that of the larger NorBERT3-large-NorICD model in 3 of the 4 settings, further supporting the impact of domain-specific pretraining for improving recall in clinical coding tasks.

Table 6 summarizes the F₁-scores for all models across the 4 evaluation scenarios. NorDeClin-BERT-large-NorICD achieved the highest F₁-score in all cases, with 0.54 for all codes multilabel, 0.81 for all codes top-5, 0.60 for the top 80% most-used codes multilabel, and 0.89 for top 80% top-5, consistently outperforming the general-domain NorBERT3-large-NorICD (0.50, 0.79, 0.58, and 0.88, respectively). The largest F₁-score margin between the large models was observed in the all codes multilabel setting (0.54 vs 0.50), highlighting the impact of domain adaptation on balancing precision and recall in complex coding tasks.

Among base-sized models, NorDeClin-BERT-base-NorICD also demonstrated strong performance with F₁-scores of 0.52, 0.79, 0.58, and 0.88, respectively. It outperformed all other base models across every scenario and matched or exceeded the performance of the larger NorBERT3-large-NorICD in all 4 settings, further validating the strength of clinical domain pretraining even in smaller architectures.

Figure 5 illustrates the attention distribution of NorDeClin-BERT-base-NorICD in a synthetic clinical text. The attention appears to be distributed relatively uniformly throughout the clinical description, suggesting that the model focuses on a comprehensive contextual understanding of the text to make predictions. Key medical terms like diaré (diarrhea), blødning (bleeding), Crohns sykdom (Crohn disease), and inflammatorisk tarmsykdom (inflammatory bowel disease) receive high attention, indicating their importance in the model’s decision-making process. The model’s interpretability is based on its attention to clinical descriptions and terminology. This approach provides valuable insights into how the model processes natural language to arrive at its predictions. It is particularly useful in understanding how the model infers ICD codes from medical text alone, mimicking the process a human expert might follow when assigning codes based on clinical narratives.

Figure 6 shows the attention distribution of NorDeClin-BERT-large-NorICD applied to the same synthetic clinical text. Like the base model, it assigns high attention weights to terms such as blødning, Crohns sykdom, and inflammatorisk tarmsykdom. However, the large model displays a slightly more focused and confident pattern, with attention concentrated more tightly around diagnostically relevant phrases. This may reflect the benefit of both greater model capacity and pretraining on the full corpus, resulting in more targeted representation learning.

Comparing the 2 models, both demonstrate strong interpretability by attending to clinically meaningful concepts. NorDeClin-BERT-large-NorICD appears to apply attention more selectively, in line with its superior classification performance. These visualizations support the idea that domain-specific pretraining not only improves predictive performance but also enhances transparency and trust in real-world clinical applications.

The distribution of attention across the text, focusing on key medical terms, suggests that the models have developed a nuanced understanding of clinical language. This method of interpretation allows us to understand which parts of the clinical narrative the models consider most relevant for predicting ICD codes. This ability to extract relevant information from various parts of the text indicates a robust and generalizable approach to ICD code prediction. It showcases the models’ capacity to process and understand clinical narratives in a way that aligns with human expert reasoning.

This interpretability analysis highlights the NorDeClin-BERT models’ potential to assist health care professionals and improve their trust by providing insight into the reasoning behind the predicted ICD codes. The models’ attention to a broad range of clinical terms and contexts suggests their potential adaptability to various types of medical narrative, which is crucial for real-world applications in diverse health care settings.

The development and evaluation of 2 variants of NorDeClin-BERT for ICD-10 code classification tasks have yielded insightful results, highlighting their capabilities and potential applications in Norwegian health care settings. Both NorDeClin-BERT-base-NorICD and NorDeClin-BERT-large-NorICD have emerged as frontrunners, demonstrating higher accuracy, precision, recall, and F₁-scores across both all codes and the top 80% most-used codes. These findings underscore the robustness and efficiency of the models in handling diverse and prevalent code classifications.

The good performance of the NorDeClin-BERT models, especially in the context of the top 80% most-used codes, suggests that these models have effectively captured the underlying patterns and nuances of the most frequent classifications in Norwegian clinical texts. This capability is critical in practical applications where prioritizing common codes can substantially enhance operational efficiency and accuracy. At the same time, the models, particularly NorDeClin-BERT-large, showed notable improvements in multilabel classification across both full and frequent-code scenarios, outperforming all baseline models in recall and F₁-score. Furthermore, the high precision of the NorDeClin-BERT models indicates their utility in scenarios where the cost of false positives is high, making them an ideal choice for critical applications in medical coding and documentation.

An important aspect of our study is the comparison of models with different sizes and architectures. NorDeClin-BERT was developed in both base and large configurations, with the base model built on the BERT-base architecture (≈110 million parameters) and the large model using a BERT-large architecture (≈340 million parameters). NorDeClin-BERT-base consistently outperformed or matched the performance of other models, including the larger general-domain NorBERT3-large model. This finding is particularly noteworthy, as it challenges the common assumption that larger models invariably lead to better performance. The success of NorDeClin-BERT-base suggests that, for specialized tasks such as ICD-10 coding in Norwegian clinical texts, a well-tuned base-size model can be highly effective and potentially more efficient in terms of computational resources and inference time.

The comparable or better performance of NorDeClin-BERT-base to larger models such as NorBERT3-large also highlights the importance of domain-specific pretraining and fine-tuning. It appears that the targeted approach of training on Norwegian clinical texts has allowed even the smaller NorDeClin-BERT variant to develop a more nuanced understanding of medical terminology and context, compensating for its reduced size. This observation has significant implications for model development in specialized domains, suggesting that carefully curated training data and domain-specific adaptation can be as important as, if not more important than, raw model size.

Furthermore, the efficiency of a smaller model like NorDeClin-BERT-base has practical advantages in clinical settings. It can be more easily deployed in environments with limited computational resources, potentially allowing for faster inference times and lower hardware requirements. This could facilitate broader adoption across various health care institutions, including those with constrained IT infrastructures.

The findings of this study have broader implications for the implementation of machine learning in Norwegian clinical settings. The NorDeClin-BERT models can substantially reduce the workload of health care professionals by automating routine coding tasks, allowing them to focus more on patient care and less on administrative duties. In addition, the enhanced accuracy and precision of these models can contribute to better patient outcomes by ensuring more accurate reporting and documentation, which, in turn, can lead to more targeted and effective patient care plans in Norwegian hospitals.

The attention-based interpretability analysis provides valuable insight into NorDeClin-BERT models’ decision-making process, which could enhance trust and adoption among health care professionals. The models’ ability to focus on relevant clinical terms when ICD codes are not present demonstrates their potential to generalize well to various clinical narratives.

Our study not only demonstrates the effectiveness of the NorDeClin-BERT models in ICD-10 coding tasks but also provides valuable insights into the trade-offs between model size, domain-specific training, and performance in specialized NLP tasks. These findings could guide future research and development in clinical NLP, potentially leading to more efficient and effective AI solutions in health care.

While this study focuses on ICD-10 coding for clinical documentation, structured coding also plays a crucial role in several other domains, including billing, epidemiological research, clinical registries, and decision support systems. In billing and insurance claims, accurate ICD-10 coding ensures proper reimbursement and minimizes administrative errors. In epidemiological studies, these codes are essential for monitoring disease prevalence and public health trends, where high recall is particularly important to ensure comprehensive case identification and minimize underreporting. Similarly, clinical registries rely on structured diagnostic coding to maintain high-quality datasets, where both precision and recall influence the completeness and reliability of registry-based research. Additionally, in clinical decision support systems, ICD-10 codes are often used to trigger alerts, inform risk assessments, or guide treatment recommendations, where high precision is crucial to avoid false-positive alerts that could contribute to alert fatigue and unnecessary interventions. While our study does not directly evaluate these applications, our findings suggest that models like NorDeClin-BERT-base-NorICD and NorDeClin-BERT-large-NorICD have the potential to improve coding accuracy in such contexts, thereby enhancing the quality of structured health data across multiple domains. Future research could explore domain-specific adaptations to optimize NLP-driven ICD-10 coding for these different use cases.

While the results of this study are promising, several limitations must be acknowledged. First, the performance of the NorDeClin-BERT models might vary with different datasets or coding systems not covered in this study, particularly those outside the gastroenterology domain. This suggests the need for wider validation across various medical specialties and health care institutions in Norway to fully understand the generalizability of the findings.

Future research should aim to address these limitations by expanding the scope of the datasets and coding systems, potentially including other medical specialties and health care institutions across Norway. Exploring the integration of the NorDeClin-BERT models into real-world clinical workflows in Norwegian hospitals and assessing their impact on efficiency and patient care outcomes would provide valuable insights into their practical utility.

Furthermore, investigating the interpretability of the NorDeClin-BERT models and user trust in automated coding systems represents a crucial research area, as these factors greatly influence the adoption of AI technologies in health care. Developing explainable AI techniques tailored to the Norwegian clinical context could further improve the transparency and trustworthiness of these models, potentially accelerating their integration into Norwegian health care systems.

This study introduced 2 versions of NorDeClin-BERT, domain-specific BERT models specifically developed for automating ICD-10 code assignments from clinical notes within the Norwegian gastroenterological domain. By benchmarking these models against both general-domain and cross-lingual BERT baselines, we addressed 3 core RQs. First (RQ1), we found that domain-specific pretraining on Norwegian clinical text consistently improved ICD-10 classification performance across all evaluation metrics, compared with general-domain Norwegian models and Swedish clinical models. Second (RQ2), we showed that scaling the model size from base to large further enhanced performance, most notably in multilabel scenarios, demonstrating that model capacity can amplify the benefits of domain adaptation. Third (RQ3), NorDeClin-BERT-base matched or outperformed NorBERT3-large in multiple scenarios, highlighting the value of targeted pretraining even with smaller architectures.

Compared with previous work on Swedish ICD-10 classification using SweDeClin-BERT [15], our models achieved competitive or superior performance, especially under strict multilabel evaluation, despite differences in language and dataset structure. To our knowledge, this is the first study to develop and evaluate BERT-based models for ICD-10 coding in the Norwegian language, setting a new benchmark for future clinical NLP research in this area.

Through detailed analysis of accuracy, precision, recall, and F₁-score, our findings demonstrate the potential of domain-specific language models to support structured clinical documentation, reduce administrative burden, and enable more accurate downstream analytics in Norwegian health care. The results highlight the NorDeClin-BERT models as superior in terms of accuracy, precision, recall, and F₁-score for both all codes and the top 80% most-used codes, consistently outperforming other BERT variants, including ScandiBERT, NorBERT3-base, NorBERT3-large, and SweDeClin-BERT. NorDeClin-BERT-large-NorICD demonstrated the highest overall performance, while NorDeClin-BERT-base-NorICD matched or exceeded the performance of larger general-purpose models in multiple scenarios. Both models demonstrate an improved ability to capture the nuances of the Norwegian language and the complexity of medical coding. The study also underscores the relevance of language-specific and domain-specific models, as evidenced by NorDeClin-BERT’s improved performance compared with models pretrained on general Scandinavian languages.

The attention-based interpretability analysis provided valuable insight into the NorDeClin-BERT models’ decision-making processes, demonstrating their ability to focus on relevant clinical terms and adapt to the presence or absence of explicit ICD codes in the text. This feature enhances the models’ potential for generalization and practical application in diverse clinical settings across Norway.