This study was approved by the Ottawa Health Science Network Research Ethics Board (OHSN-REB Protocol number 20240543-01H).

We performed a comparative observational prospective study at a single tertiary-care nuclear medicine division, evaluating impressions of positron emission tomography (PET)/ computed tomography (CT) reports generated between September 30, 2024 and December 29, 2024. The study compared trainee-generated impressions with impressions generated by an LLM (ChatGPT-4o) from identical trainee-generated report stems before attending physician review and final report sign-off.

We gathered 200 PET/CT reports and deidentified them. Each report consisted of a clinical history, technical aspects, key findings (stem), and a final impression. The trainees had access to clinical notes, pathology reports, and prior imaging findings when drafting their reports and impressions during routine reporting, prior to review by attendings. The first 3 sections of the 200 PET/CT reports were completed by trainees. Half of the cases had the impressions generated by the trainees, whereas the remaining half (100 cases) had ChatGPT-4o–generated impressions.

The AI-generated report was produced using a standardized prompt provided to ChatGPT-4o. A project was defined within the application, and relevant templates were uploaded along with a structured prompt. The prompt specified how the impression should be organized based on the findings described in the stem, reason for referral, and any additional information available at the time of reporting. All scans were selected consecutively during routine reporting. The selection included a random mixture of PET/CT to reflect a representative sample of clinical practice, including F18-FDG (fluorine-18 fluorodeoxyglucose), Ga68-DOTATATE, and Ga68-PSMA (gallium-68 prostate-specific membrane antigen) PET/CT images. All identifiers were removed, and alphanumeric codes were assigned and aggregated into a Microsoft Excel file formatted to collect impression quality scores by the reviewers.

To limit output variability and ensure that the generated content was clinically relevant and appropriate, we developed a project template for ChatGPT-4o. The template instructed the model to create an organized and structured impression by (1) synthesizing the history, requisition, and PET/CT findings into a concise but complete impression (≤3 sections); (2) using a certainty lexicon mapped to probability categories (very low <10%, low ~25%, intermediate ~50%, high ~75%, and highest >90%); (3) including a diagnosis or differential; (4) suggesting appropriate next steps (eg, additional imaging, histopathology correlation, clinical consult, or interval follow-up); and (5) providing staging or response assessment where applicable (eg, complete response, partial response, Deauville score for lymphoma, TNM, molecular imaging TNM, or Krenning score where relevant). We performed a pilot study with a limited number of random cases prompted on lymphoma, lung cancer, and Ga68-DOTATATE and refined the instructions to ChatGPT-4o on how to create better, more comprehensive, yet short impressions before unifying all prompts and instructions into a single project (“Impression for PET-CT Reports”). The final prompt and certainty lexicon were held constant across cases. These categories were selected to reflect commonly used probability-based terminology in radiology and nuclear medicine reporting. The questionnaire and a list of certainty words were provided to raters (Multimedia Appendix 1).

Three board-certified nuclear medicine physicians (coded as EL, CW, and FA-R) independently rated each impression. Raters were blinded to the report origin (AI vs trainee) and to the trainee subgroup (residents vs fellows). Two raters reviewed 65 impressions, and 1 rater reviewed 70 impressions, with sufficient overlap to conduct interrater agreement analysis. Raters’ evaluations were solely based on the information in the first 3 sections of the report, and no imaging or clinical data were reviewed.

Each rater used a predefined questionnaire encoded in an Excel file to rate the impression regarding the following aspects: (1) correctness or relevance of the impression (binary scale scored −1, 0, 1; reported as proportion correct); (2) presence of at least one certainty word (yes or no); (3) certainty category (0‐5 scale: 0=none, 1=very low to 5=highest); (4) Likert-scale (1-5) rating questions regarding clarity, completeness, organization, and overall satisfaction; (5) presence of an action plan and type (further imaging, histopathology, close follow-up, and clinical consult; multiple responses permitted); and (6) free-text comments for thematic analysis. The questionnaire was internally reviewed and pilot-tested using a limited set of PET/CT cases before formal evaluation. Correctness was defined as the factual accuracy of the impression relative to the findings section of the same report, ensuring that the interpretation was consistent with the imaging descriptions. Relevance was defined as the clinical pertinence and prioritization of information within the impression, reflecting whether it addressed key diagnostic questions and actionable conclusions. Both attributes were assessed by expert reviewers using a structured 5-point Likert questionnaire based solely on the written report content. Similarly, clarity was defined as the degree to which the impression communicated its diagnostic message in a precise, unambiguous, and easily understandable manner. Reviewers assessed clarity using a 5-point Likert scale (1=very unclear to 5=very clear), considering sentence structure, readability, terminology, and logical flow of the impression text. Raters also evaluated the action plan based on information provided in the stem of the report and the referral note. If the action plan provided was inappropriate, the rater marked it appropriately and was able to comment on it in the open-ended comment section. Completeness was defined as the degree to which the impression summarized all major findings and directly addressed the primary clinical question. Reviewers rated completeness subjectively on a 5-point Likert scale, considering whether any omission could alter clinical interpretation or management. Examples of incomplete impressions included failure to mention key lesions or to provide a definitive diagnostic statement when sufficient information was available. Finally, the rater identifier was also recorded for interrater analysis.

Interrater reliability was assessed using Fleiss κ on a common sample, with κ=0.61 to 0.80 interpreted as substantial agreement. Continuous outcomes (Likert means) were compared using independent-samples 2-tailed t tests (AI vs trainees), within subgroup comparisons (AI vs residents vs fellows), and using t tests and/or one-way ANOVA as appropriate with related P values. Categorical outcomes were compared with chi-square tests and reported with P values and Cramér V where applicable. Effect sizes for continuous outcomes were expressed as Cohen d (0.2 for small, 0.5 for medium, and 0.8 for large). Statistical significance was set at α=.05 (2-sided). All descriptive and comparative statistical analyses (means, SDs, t tests, and chi-square tests) were computed in Microsoft Excel. Advanced analyses, including one-way ANOVA with Tukey Honestly Significant Difference post hoc testing, were performed in GraphPad Prism (version X). Cohen d effect sizes were calculated manually from group means and pooled SDs using standard formulae.

We analyzed 200 impressions, comprising 100 (50%) AI-generated and 100 (50%) trainee-generated impressions. Of the 100 scans, 90 (90%) were F18-FDG, 7 (7%) were Ga68-DOTATATE, and 3 (3%) were F18-DCFPyL (prostate-specific membrane antigen) PET/CT scans. Of the 100 cases, 34 (34%) were lung cancer, 19 (19%) lymphoma, 8 (8%) breast cancer, 8 (8%) melanoma, 7 (7%) neuroendocrine tumor, 7 (7%) colon cancer, 5 (5%) esophageal cancer, 3 (3%) prostate cancer, 3 (3%) multiple myeloma, 3 (3%) nasopharyngeal cancer, 2 (2%) pancreatic cancer, and 1 (1%) sarcoma case. Among the trainee-generated reports, 26 (26%) were authored by 2 Canadian radiology residents (referred to as “residents”), who were native English speakers and participated during their nuclear medicine rotations in postgraduate year 4 and year 5 with limited prior nuclear medicine experience. The remaining 74 (74%) reports were authored by 4 international nuclear medicine fellows (referred to as “fellows”), who were nonnative English speakers but had prior experience in PET/CT reporting during their nuclear medicine or radiology residency training, or had completed at least 1 year of nuclear medicine fellowship in their home country. Of these fellows, 2 were in year 1 and 2 were in year 2 of their current training. Rater contribution was balanced across the 200 total reviews: EL reviewed 65 (32.5%), CW reviewed 65 (32.5%), and FA-R reviewed 70 (35%). Interrater reliability was performed over 20 cases with the overlap cases used for interrater reliability assessment, including 12 of 20 (60%) cases, yielding Fleiss “κ” value of 0.71, which indicates substantial agreement among raters. Although the overlap sample was limited, this level of agreement suggests consistent application of the rating criteria.

The results of this study highlight the advantages of AI-generated impressions in nuclear medicine compared to trainee-generated impressions, aligning with contemporary literature on AI-assisted diagnostic imaging [9-13]. Importantly, the observed improvements in completeness and clarity reflect enhanced synthesis of provided information rather than independent image-based diagnostic reasoning. AI-generated impressions in this cohort demonstrated superior completeness and a more consistent use of diagnostic certainty terminology—the 2 domains previously identified as sources of variability and potential clinical error in nuclear medicine reporting. Although statistically significant, these differences were associated with small effect sizes, suggesting modest clinical impact that should be interpreted with caution. AI-assisted reporting depends on the quality and completeness of the input data, and omissions in the report stem may be propagated rather than corrected. The AI model was provided with structured report inputs, which differ from real-world trainee workflows and may have contributed to improved completeness and standardized language. Standardized diagnostic language is instrumental in communicating risk, guiding downstream decisions, and reducing the likelihood of misinterpretation by referring physicians [14-17].

Clarity and correctness were found to be comparable between AI-generated and trainee-generated impressions, a finding echoed in studies evaluating the quality of AI-generated reports against human experts in radiology and nuclear medicine [18]. While AI systems deliver structured and highly reproducible outputs, human-generated reports, especially those authored by more experienced trainees, still provide nuanced clinical context—an aspect where AI sometimes underperforms due to its reliance on template-driven logic and a lack of experience-driven intuition [19,20]. The lack of brevity in this study may be mitigated through prompt optimization, including explicit length constraints and structured templates to balance completeness with conciseness.

Notably, a subgroup analysis in this study indicated that international trainees with previous nuclear medicine training outperformed radiology residents who were native English speakers but had less experience in nuclear medicine PET/CT imaging in terms of report completeness and clarity, suggesting that exposure remains a critical lever for report quality. AI may be beneficial for less experienced trainees, helping them improve quickly and reducing mistakes or missing information by supporting their learning.

Other studies also support this, with AI integration shown to boost efficiency, reduce addendum rates, and streamline patient triage [21,22]. However, there is a need for continued oversight and physician presence in AI-generated reports to mitigate potentially negative management impacts. Future implementations should ensure that AI report provenance is clear, outputs are subject to faculty sign-off, and safeguards are in place for rare or complex cases. Prospective, multicenter studies should also evaluate clinical outcomes related to AI-assisted reporting, including diagnostic yield, time to diagnosis, and medicolegal parameters [11,13,23].

In this blinded comparison of AI-generated and trainee-generated PET/CT impressions, AI achieved noninferior correctness and clarity while demonstrating advantages in completeness and consistent use of certainty terminology. Several evaluated outcomes (eg, clarity, completeness, and satisfaction) are inherently subjective, and although interrater agreement was substantial, this subjectivity may influence the interpretation of performance differences. These findings suggest that structured AI outputs can address 2 common challenges in trainee reporting, including omissions and inconsistent phrasing without sacrificing accuracy. The higher frequency and level of certainty terms in AI impressions are notable. The use of certainty modifiers carries meaningful clinical implications in diagnostic communication.

In this study, AI-generated impressions contained certainty words more frequently and tended to express higher degrees of diagnostic confidence compared with trainee reports (Table 1; P=.02). Excessive omission of such terms in trainee reports can lead to ambiguous or overly cautious interpretations, potentially limiting clinical decisiveness or delaying management decisions. Conversely, overly assertive wording without appropriate qualification may create false diagnostic confidence and potentially contribute to inappropriate clinical decision-making. Maintaining calibrated, probability-based terminology (eg, “likely ~75%” and “consistent with >90%”) helps align radiologic communication with clinical reasoning and supports reproducible decision-making.

Our findings suggest that structured AI phrasing may enhance clarity and interpretive confidence, but appropriate contextual use of certainty words remains essential to balance precision with caution in diagnostic reporting. Standardized diagnostic language helps communicate probability and clinical confidence, which is central to downstream decision-making [24]. Our results align with literature describing AI’s capacity to reduce variability and improve report standardization in radiology [24]. Recent work has also emphasized the importance of robust evaluation frameworks for LLM outputs in clinical contexts [25]. Emerging evidence also supports the role of LLMs in imaging-related clinical interpretation workflows [26]. By contrast, trainees’ strengths lie in case-specific details and adaptive reasoning, which are qualities that remain essential for complex or atypical presentations. Subgroup results provide important context that international trainees (with ≥1 y of nuclear medicine training) outperformed Canadian radiology residents (with limited nuclear medicine exposure) in completeness and, numerically, clarity. This supports the interpretation that training duration and exposure are key determinants of report quality. It also underscores how AI might be most beneficial for earlier-stage trainees or in rotations where exposure is limited, by providing consistent scaffolding and a didactic template. Clinically, more correctness (92%) and complete impressions (97%) in AI-generated impressions, along with explicit action plans (67/81, 82.7%) compared to trainee-generated reports (91%, 92%, and 59%, respectively), would reduce ambiguity and help standardize care pathways by prompting appropriate next steps (eg, histopathology or additional imaging). Nevertheless, 3% of reviews signaled potential negative impacts in AI-generated impressions though still below the negative impacts of trainee-generated impressions, highlighting the need for human oversight. AI-generated impressions may use generic lexicon or be too wordy, but giving the AI more specific instructions or using locally designed templates can help make AI outputs more precise and relevant.

In conclusion, the findings of this study support a model in which AI further improves the quality of the impression section of the PET/CT report, particularly among less experienced trainees with limited expertise in nuclear medicine reporting. Carefully designed AI templates and locally tailored instructions may provide a practical pathway to improving impression quality, supporting trainee education, and ultimately enhancing patient care. The use of AI in diagnostic reporting also raises ethical considerations regarding accountability, as the ultimate responsibility for interpretation and clinical decisions remains with the reporting physician.

This single-center study used a moderate sample size (200 impressions) and a single AI model (ChatGPT-4o) with a fixed prompt and lexicon. Results may differ with other models, other sorts of prompt designs, templates, or clinical settings. Although raters were blinded, stylistic cues might have partially unblinded the source. The case mix was limited to PET/CT studies in one institution and did not include outcome-linked end points; thus, clinical impact is inferred rather than demonstrated. In this study, the trainees had access to full information (pathology, prior images, treatments, clinical information, etc), while AI and raters assessed impressions only from report stems. This constitutes a limitation and should be acknowledged. The difference in information access may have influenced comparative performance and should be considered when interpreting the results. A key limitation of this study is that AI-generated impressions were based solely on the written report components (clinical history, technical details, and findings/stem) and did not incorporate direct image interpretation. Therefore, this study evaluates the linguistic quality, structure, and consistency of report impressions rather than true diagnostic accuracy relative to imaging data. As such, the findings should be interpreted as reflecting improvements in report formulation and usability rather than the independent diagnostic capability of AI systems.

Interrater reliability and the possibility of improving agreement could be further explored through studies investigating sources of interrater disagreement, including whether specific lesion types or clinical scenarios contribute to variability in scoring. Finally, subgroup analyses (residents vs fellows trainees) may be influenced by unmeasured confounders (eg, prior report volume, language proficiency, or subspecialty focus).

Prospective, multicenter trials should evaluate workflow integration (time-to-report, addendum rates), downstream clinical outcomes (diagnostic yield of recommended actions, time-to-treatment), and medicolegal dimensions (consistency, error rates, and escalation triggers). Research should test adaptive AI systems that learn local style guides and incorporate structured reporting templates and certainty lexicons. Educational studies should determine how AI assistance influences trainee learning curves, autonomy, and feedback quality. Transparent frameworks are needed to define responsibilities, auditing, and disclosure of AI assistance in clinical documentation. Finally, emerging approaches, such as physics-informed AI that integrates quantitative imaging biomarkers (eg, standardized uptake values) with language models, may further enhance the clinical utility of AI-assisted reporting. Given the similar reporting structure across nuclear medicine studies, this approach may extend to other areas; however, dedicated validation in different modalities and clinical settings is warranted.

AI-generated PET/CT impressions, based on the provided report text (without direct image input), were comparable to trainee impressions in correctness and clarity and were superior in completeness, consistent use of certainty terminology, and in providing action plans when appropriate. International fellow trainees outperformed Canadian residents, supporting the role of prior nuclear medicine experience in report quality. Together, these results indicate that AI can function as a complementary tool for standardizing language, reinforcing completeness, and prompting explicit next steps in the impression of the report, while faculty oversight preserves clinical correctness and safeguards against generic or misleading statements. Despite improvements in completeness and structure, the similar rate of potential negative medicolegal impact suggests no clear advantage in core diagnostic safety in this academic setting; however, variability across practice environments means even small reductions may be meaningful, reinforcing AI as an assistive rather than replacement tool. Adoption pathways should prioritize education, local customization of prompts and lexicons, and thorough evaluation of clinical and operational outcomes.