The discharge summary is a critical clinical document that communicates a patient’s care in hospital, key clinical events, and plans for follow-up care. It typically includes patient demographics, hospital information, diagnoses, a clinical summary, medications, allergies, and pathology results. Currently, the responsibility for compiling the discharge summary often falls on a single clinician. This presents several challenges. First, inpatient care is often delivered by a multidisciplinary team of health care professionals [1], making it difficult for any single clinician to have complete oversight of all relevant clinical events and decisions. Second, the required information is dispersed across multiple documents within the electronic health record (EHR), including admission clerking notes, ward round entries, operation notes, and results (eg, pathology and radiology), making the process of locating and synthesizing key data time-consuming [2]. Third, EHR data are predominantly unstructured, with approximately 80% existing as free text [3], further complicating retrieval and summary generation. Finally, the volume of EHR data is continually increasing [4], intensifying the burden on clinicians tasked with producing accurate and comprehensive discharge summaries under time pressure.

Generative artificial intelligence (AI) refers to a class of AI that can create new content such as text [5] based on patterns learned from large datasets. A specific type of generative AI is the large language model (LLM) [6], which is trained to understand and generate humanlike text. In a clinical setting, LLMs have the potential to streamline documentation, automate routine tasks, and support decision-making within EHRs. A commonly used LLM is the generative pretrained transformer (GPT), which was first released by OpenAI in June 2018 [7] and has been continuously optimized into newer versions, including GPT-4, which we used in this study [8]. GPT has demonstrated the ability to generate humanlike text and can be adapted for specific tasks such as producing discharge summaries [9-18].

To leverage pretrained LLMs in clinical settings, prompt engineering has become increasingly central to research in developing clinical applications that minimize hallucination rate, preserve factual accuracy, and produce outputs for clinical use [19]. Prompt engineering is the deliberate design of inputs to generative models to align the output with the intended clinical application without the need for retraining [20,21]. A well-designed prompt provides structured and explicit cues to the model, facilitating accurate and efficient task completion. A range of different prompt patterns have been developed and applied in the context of text generation and summarization, and these techniques are increasingly being applied in clinical settings [22].

The use of LLMs to automate discharge summary generation has been examined using both synthetic or publicly available datasets [14-17] and real-world EHR data [9-13,18,23-25]. Among studies using real hospital data, several imposed constraints on the clinical cohort. Li et al [9] and Kim et al [12] restricted their populations to specific patient conditions [18] or to settings such as the emergency department (ED) [11] or narrowed inclusion by limiting the length of stay [10,13]. Ganzinger et al [23] relied on structured data that had to be manually extracted from semistructured and unstructured EHRs to construct the prompt context, underscoring the challenges of working with heterogenous clinical records.

This study aimed to develop a template-based prompting system that can produce clinically acceptable discharge summaries, specifically the “clinical summary” and “plan and requested action” sections, from unstructured, routinely collected electronic patient records. The system was iteratively co-designed by data scientists, engineers, and physicians. The system retrieved free-text clinical notes related to an inpatient hospital encounter and used structured template prompts to guide OpenAI’s GPT-4 in synthesizing the clinical documentation and generating a discharge summary that conformed to the UK Royal College of Physicians’ (London) 2021 guidelines [26] and National Health Service England’s information standard DAPB4042 [27]. The clinical acceptability of the generated summaries was evaluated using an assessment form completed by a clinical team of 4 resident physicians and included the primary outcome (physicians’ global confidence rating) and secondary outcomes (accuracy and completeness, clinical significance of incorrect or missing content, formatting, readability, and sociodemographic bias).

We present the development of a prompt-templated pipeline for generating discharge summaries using OpenAI’s GPT-4 model [8] from routinely collected patient EHRs. From our prompt engineering iterative process, our final prompt used examples of high-quality input-output pairs for few-shot learning. The utility and clinical safety of the generated discharge summaries was evaluated by resident physicians. Our results demonstrate the potential effectiveness of this system, with evaluations by resident physicians showing a majority (8/9, 89%) of positive ratings on the primary outcome (global confidence rating), as well as on secondary outcomes (readability, formatting, sociodemographic bias, and potential clinical harm). However, rigorous statistical evaluation in a larger, adequately powered sample is needed [23]. As discussed below, this study presents a series of innovations relative to the work previously done in this field.

First, by harnessing the Imperial SDE [28], the data used to generate discharge summaries in this study came from unstructured, real-world EHRs. This constitutes a significant advancement in the field, which has largely been limited to synthetic or publicly available datasets or structured data extracted from the EHR [14-17]. Moreover, the prompt was designed and refined by a multidisciplinary team of resident physicians, data engineers, and data scientists through an iterative approach informed by the Royal College of Physicians discharge summary guidelines [27]. The physicians’ domain expertise and active involvement ensured that the system was not only performant but also aligned with the practical realities of clinical workflows.

Second, although the final evaluation was limited to a small sample (n=9), we found potentially promising results for the clinical acceptability of the generated outputs, aligning with previous research [9,13]. For example, 100% (9/9) of our discharge summaries were rated as clear and concise relative to lower ratings in previous studies (eg, Cai et al [29] observed that only 39% of the summaries obtained a full score on readability). The high readability score could be attributed to aspects of our pipeline, such as the low value of the top_p parameter, which reduces the use of excessive text decorations and ensures that the summary remains concise and focused on the relevant medical information. Moreover, the prompt specified that medical acronyms in the text should be translated to their expanded term following the Unified Medical Language System, which could reduce the ambiguity of medical acronyms [30].

Third, in addition to the prompt design, we showed that the selection of input data for the prompt template is important and can affect the quality of the discharge summary output. This was previously shown by Hartman et al [31], where progress notes and consults were the primary source of content selected for generating a summary of a patient’s hospital stay, with other documents being removed. We carried out a service evaluation with clinical experts to identify the key inputs needed for effective discharge summary generation. These included admission clerking, ward round notes, specialty reviews, operation notes, and diagnostic results (eg, pathology and radiology). Due to a proportion of the clinical notes in the EHR being of unknown type, such as blank notes (a text file without any preformatting or title), clinical notes were filtered by excluding those deemed irrelevant for drafting a discharge summary, such as nursing notes and board round notes. By using this cleaner set of input documents in iteration 2, improvements were observed in the discharge summary evaluation for the primary outcome compared to iteration 1 (28% more responses were positive for the global confidence rating), as well as some secondary outcomes (eg, the “clinical summary” section improved by 4% on completeness and 5% on accuracy, and readability improved by 37%). However, it is worth noting that these improvements may also be attributed to any changes to the prompt across iterations.

This study presents several avenues for future research. First, the evaluation of LLM-generated outputs can be further expanded. For example, for a proportion of cases in both iteration 2 and the test dataset (9/37, 24%; 3/9, 33%; Multimedia Appendix 1), resident physicians stated that they needed to consult with the team that treated the patient due to insufficient data in the clinical notes. This corresponds with the work by Ando et al [24], who found that 39% of the discharge summary information was sourced externally to the EHR, with up to 11% derived from the clinician’s memory. To further understand the need for a human in the loop, additional metrics such as gap distance could be introduced in the evaluation form, as suggested by Cai et al [29]. In addition to qualitative evaluation, quantitative methods such as the Recall-Oriented Understudy for Gisting Evaluation score [32], bidirectional encoder representations from transformers score [33], and Flesch-Kincaid readability ease [34] provide an approach for standardized evaluation of LLM-generated outputs.

To move toward the safe implementation of our pipeline into a real-world clinical setting, several challenges, such as model bias and fairness [35], would need to be addressed, as outlined in previous studies [36]. Moreover, a better understanding of the clinical scenarios where the tool could be implemented successfully is required. Case complexity often results in a higher number of documents, which poses a challenge due to the limited effective context window of LLMs [37]. To address this, future research should explore methods to improve the selective preprocessing of input data. Research could also stratify cases by factors indicative of case complexity, such as treatment specialty, care transitions, or length of stay. We explored some of these in our sensitivity analysis, but a larger sample would be necessary to inform data-driven clinical guidelines.

Safe implementation in practice would also require careful attention to data privacy, transparency, and model governance. In this study, data were deidentified, and the model was deployed through Azure OpenAI within a managed virtual network inside the Imperial SDE, providing network isolation and controlled access. Prompts and outputs were processed only for inference and were not used to train the underlying foundation models. If implemented in routine clinical practice, similar technical safeguards would need to be complemented by transparent documentation and clear governance around proprietary model updates. As proprietary LLMs such as the GPT series are typically subject to vendor-driven version changes, mechanisms for version control, auditability, and ongoing performance monitoring would be necessary to ensure that model updates or changes in clinical data environments do not adversely affect reliability, reproducibility, or patient safety.

There are several limitations worth considering. First, missing data present a challenge for LLMs [24]. Task completion is rarely documented by physicians in EHRs, leaving the model to infer which actions require follow-up in discharge plans, often resulting in inaccuracies. Similarly, some remote specialist or senior input, such as phone consultations, is frequently underdocumented or even omitted. Moreover, due to anonymization preprocessing in our study, key details such as time stamps and locations were removed. This hindered the model’s ability to sequence clinical events chronologically. However, this issue would be mitigated in a real-world clinical setting, where the data are not anonymized. Second, different specialties often use inconsistent formats, and plans can sometimes conflict, such as between renal and cardiology teams. In some cases, the model struggled to generate unified discharge plans. Moreover, medical abbreviations can be ambiguous, and their meaning may vary based on context and specialty (eg, “PT” could refer to physiotherapy, prothrombin time, or patient). Directly integrating the model with a terminology or abbreviation database such as the Unified Medical Language System could help improve its interpretative accuracy. Third, given that general practitioners and patients are the key readers of discharge summaries, their evaluation of the GPT-generated summaries would also be valuable. Fourth, the small sample of discharge summaries limits our ability to formally assess the clinical acceptability of the generated summaries or potential sources of bias. In addition, cases were manually selected by clinicians who had directly cared for the patients; therefore, the sample was not representative of the broader patient population and may be subject to selection bias. For the purposes of this study, which primarily aimed to codevelop an effective prompt, resident physicians were asked to select cases with a range of lengths of stay to capture varying levels of clinical complexity. In addition, interrater reliability was not assessed because each case was evaluated by a single physician. Future work should evaluate a larger set of generated discharge summaries across a representative sample and include assessment of interrater reliability and whether specific patient, clinical, or contextual factors systematically influence the quality. However, this would be very resource intensive due to the time-consuming aspect of discharge summary evaluation. For example, in our study, clinician evaluation took approximately 10 minutes per case. As proposed by Gero et al [38], LLM self-verification tools could be used to show the source of the input data used in the discharge summary generation and, thus, shorten the time required for physicians to review the input notes.