Following the Joanna Briggs Institute (JBI) framework for scoping reviews [9], and reporting in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews; Multimedia Appendix 1) [10], this review maps the available evidence on computational methods for automating CPG development across four pillars: (1) retrieval of relevant literature, (2) extraction of structured information from guideline documents, (3) encoding of guidelines into machine-interpretable formats, and (4) automated quality evaluation. The primary objective was to identify and chart the range of computational methods that have been empirically tested for automating guideline development, the tasks they address, and the outcome measures used to evaluate them, and to map the distribution of that evidence across the 4 pillars in order to expose knowledge gaps and areas of fragmentation. Consistent with scoping review methodology, the review charts what evidence exists and how it has been evaluated; it does not pool results, compare methods against one another, or critically appraise the quality of individual studies [9]. An OSF project for this review was created on March 16, 2026 [11]. The review was not prospectively registered. The project record described the review’s scope but did not specify databases, search strategies, or eligibility criteria in advance, and a registration documenting the methods as conducted was completed retrospectively in August 2026 [12]. The scope of the review evolved during conduct: it was initially framed around 3 pillars, and retrieval was subsequently added as a fourth.

We searched PubMed and Scopus on August 10, 2026, for studies published between January 1, 2001, and December 31, 2025. The start date reflects the emergence of shareable computer-interpretable guideline (CIG) formalisms [13].

The search combined 2 sets of terms with OR. The first comprised phrases naming guideline formalisms, appraisal instruments, and automation tasks (eg, “computer-interpretable guideline,” “GLIF3,” “APPRAISE-RS,” and “AGREE II scores”). The second comprised paired terms, each combining a guideline concept with an automation, computational, or process concept (eg, “large language model” AND “clinical practice guideline”; “information extraction” AND “clinical guideline”). Terms were drawn from the vocabulary of studies known to the review team and refined iteratively; the search was not peer-reviewed. Full strategies for both databases are given in Multimedia Appendix 1.

Reference lists of included studies were screened and forward citation searching performed, identifying 8 further records. Gray literature, preprints, and dissertations were not searched. Records were imported into Covidence (Future Evidence Foundation), where 898 duplicates were removed before screening.

Studies were eligible if the automated component acted on a guideline development task as defined below, reported original research with empirical testing, and provided sufficient methodological detail for assessment.

Studies were excluded if they addressed guideline implementation or adherence only, were opinion pieces without primary data, or were not published in English.

For the purposes of this review, CPG development refers to the processes by which a guideline is produced or updated: identification and screening of evidence, extraction and structuring of evidence and recommendation content, representation of recommendation logic in a machine-interpretable form, and appraisal of the resulting evidence or recommendations. It excludes processes downstream of a completed guideline, namely dissemination, point-of-care execution of an existing guideline in a decision support system, and measurement of clinician adherence. A study was eligible only if the automated component acted on a guideline development task as defined above. Where a computable artifact was authored as part of guideline production, it was treated as development (encoding); where an existing guideline was executed against patient data at the point of care, it was treated as implementation and excluded. A living guideline was defined as a guideline maintained through continuous or scheduled surveillance of new evidence with incremental updating of affected recommendations, rather than through periodic full revision. Studies were assigned to a pillar according to the task the automated component performed: retrieval, where the component identified or screened candidate evidence; extraction, where it parsed narrative text into structured elements; encoding, where it produced or verified a formal, machine-interpretable representation; and evaluation, where it scored or appraised evidence or recommendation quality against an instrument such as GRADE or AGREE II. Studies whose components spanned more than one task were assigned to every applicable pillar, so pillar counts are not mutually exclusive.

Titles and abstracts were screened independently and in duplicate by 2 reviewers in Covidence systematic review software, followed by independent duplicate full-text screening against the same criteria. Before formal screening, both reviewers piloted the eligibility criteria on a sample of records and met to resolve interpretive differences; formal screening proceeded once agreement was judged acceptable. Disagreements at either stage were resolved by discussion, with a third reviewer available for arbitration where consensus could not be reached. Records retained after full-text screening were assessed against the operational definitions below. Two reports were excluded at this stage: one was a review and analysis of evaluation criteria rather than original research with empirical testing [14], and one met all other criteria but was published in Spanish [15].

This scoping review synthesized data from previously published, publicly available studies. It involved no human participants or animals and no access to individual-level or identifiable patient data. Institutional review board approval and informed consent were therefore not required, consistent with the policies of King Abdulaziz University. No primary data were collected. All included studies are cited in full, and no reproduction of copyrighted material beyond fair academic use was undertaken.

The search identified 2577 records: 998 from PubMed, 1571 from Scopus, and 8 through backward and forward citation searching. After removal of 898 duplicate records, 1679 proceeded to title and abstract screening. Two reviewers independently assessed records against the predefined eligibility criteria, with disagreements resolved as described in the “Methods” section. Of these, 1450 records were excluded.

The remaining 229 records were sought for full-text retrieval; 25 could not be obtained, leaving 204 reports for full-text eligibility assessment. Full-text exclusions totaled 157: 61 addressed evidence synthesis or not targeting CPG development automation (R1); 41 focused on clinical decision support or point-of-care use without automating any guideline development process (R2); 29 lacked original research on CPG automation and were reviews, opinions, commentaries, or protocols (R3); 20 did not present a transferable computational method for CPG development, including disease-specific tools or essentially manual approaches (R4); 5 provided insufficient methodological detail to assess the computational approach (R5); and 1 met all other criteria but was not published in English (R6).

The final corpus of 47 studies [17-63] spans 2001-2025 and is characterized in Table 1 and detailed study by study in Table 2. Performance metrics by automation task are summarized in Table 3. The selection process is summarized in Figure 1. Publications showed a marked increase after 2015, with a specific surge in LLM-related work during 2024-2025. The relatively high attrition from 2577 records to 47 inclusions reflects the breadth of the search strategy alongside the specificity of the scope: studies empirically testing computational methods for automating the guideline development process itself—as opposed to using or implementing existing guidelines—represent a considerably smaller subset of the biomedical literature.

Six studies investigated the automation of literature search and screening, revealing a trade-off between efficiency and comprehensive recall [17,37,48,54,55,60]. Query expansion methods improved literature retrieval, achieving 80% recall in identifying relevant citations [17]. Deep learning approaches demonstrated additional promise: Lin et al [55] proposed the Paper title, Abstract, and Journal (PAJO) model, combining PubMedBERT with journal-level features for automated literature screening in CPG development, achieving an area under the receiver operating characteristic curve (AUC) of 91.84% on a neck pain evidence corpus and outperforming text classification baselines by 1.91% in F₁-score.

However, the transition to semantic and generative search tools highlighted limitations in reliability. While semantic processing tools (eg, SemRep [US National Library of Medicine]) achieved high specificity (98%) and precision (88%) in answering clinical questions, they suffered from low recall (40%), rendering them more suitable for filtering than for the comprehensive searching required in systematic reviews [37]. Furthermore, recent evaluations of generative AI indicated that while LLMs can successfully generate search syntax, they currently fail to independently execute retrieval or screening with the rigorous reliability necessary for systematic reviews [60].

Fifteen studies addressed the extraction of structured data from narrative text, showing a clear performance divide between rule-based systems and generative models [17-19,25,33,35,37,38,43,46,51,54,59,60,63]. Previously, ontology-driven approaches applied linguistic pattern extraction to map breast cancer guideline text to formal representations (Asbru/MHB [Many-Headed Bridge]), achieving recall of 72%-80% and demonstrating the feasibility of semiautomated formalization from narrative clinical text [35].

Early rule-based NLP systems demonstrated high reliability in specific domains; systems using Unstructured Information Management Architecture (UIMA) and MetaMap (US National Library of Medicine) achieved an F-measure of 81.96% for extracting clinical actions and >70% for fine-grained attributes such as dosage [19]. Hybrid approaches combining NLP with upper-level ontologies (eg, EliXR) successfully covered 86.2% of eligibility criteria in free-text guidelines [46].

In contrast, modern LLM approaches yielded mixed results. While capable of producing structurally coherent recommendations, LLMs demonstrated an accuracy gap between overall performance (77% accuracy) and complex evaluation items (29%-40% accuracy) [58]. Critical evaluations emphasized that LLMs struggle with extraction accuracy compared to human performance, necessitating human-in-the-loop verification to prevent data hallucinations [60].

Seventeen studies explored automated quality assessment [23-26,31,39,43,45,47,48,50,54,57,58,61-63].

This scoping review identified 47 studies addressing the automation of clinical practice guideline development, spanning 2001-2025 across 4 pillars [17-63]. In the retrieval pillar (n=6) [17,37,48,54,55,60], the transition from keyword-based filtering to transformer-based classifiers represents the most consequential methodological shift, with deep learning approaches achieving AUC values exceeding 90% on structured evidence corpora [55]. In the extraction pillar (n=15) [17-19,25,33,35,37,38,43,46,51,54,59,60,63], ontology-driven linguistic pattern systems established the baseline [35], while more recent NLP pipelines have extended coverage to complex conditional and temporal constructs; however, recall rates for free-text guideline sentences indicate that human oversight remains necessary [59,60]. In the encoding pillar (n=26) [20-22,27-34,36,38-44,49,51-53,56,62,63], 2 decades of fragmented formalism development have been followed by recent work oriented toward FHIR-based representation. Two studies in this corpus [56,62] demonstrate FHIR implementation guides that produce interoperable, incrementally updatable guideline artifacts compatible with existing health information systems. Whether this represents convergence across the field or an emerging direction cannot be determined from 2 studies [56,62] published within 2 years of each other. In the evaluation pillar (n=17) [23-26,31,39,43,45,47,48,50,54,57,58,61-63], LLM-based tools produced structurally complete AGREE II and GRADE appraisals but assigned higher absolute scores than human appraisers in both studies that made the comparison, with wide limits of agreement [26,61]. This supports their use for triage and prioritization rather than as standalone quality arbiters. Across all pillars, no study has yet demonstrated a fully integrated, validated end-to-end pipeline connecting automated retrieval, extraction, encoding, and evaluation in a production clinical environment. This fragmentation represents the defining knowledge gap of the field.

A caution applies to every performance figure reported in this review. The included studies used noncommensurable outcome measures—precision, recall, AUC, F-measure, accuracy, Cohen κ, intraclass correlation coefficient, logical coverage, time saved, and clinician satisfaction—assessed against different reference standards, on corpora ranging from 28 guidelines to 89,940 abstracts, and under evaluation conditions ranging from retrospective benchmarking to simulated cases. These values cannot be pooled, ranked, or read as a common scale of maturity. It is more informative to distinguish the dimensions on which evidence exists from those on which it does not. Across the corpus, technical feasibility is well established in all 4 pillars and accuracy in controlled settings is documented for many individual tasks. Reproducibility is reported in only a small number of studies. Interoperability is addressed by the recent FHIR work [56,62]. Clinical safety, workflow integration, scalability, cost-effectiveness, and real-world adoption are essentially unevidenced: no included study reported an evaluation of an automation component embedded in an operating guideline program. The most defensible reading of this literature is therefore that automation demonstrates clear promise for bounded, structured tasks within guideline development, while robust evidence for end-to-end, production-scale living guideline pipelines remains limited.

This scoping review establishes that automated technologies for guideline development have progressed from theoretical proof-of-concept to practical, augmentative tools. A clear dichotomy exists in the current state of the art: rule-based and ontology-driven methods offer the rigorous reliability required for clinical safety and logic verification[20,21,49], while emerging LLMs offer the processing speed needed to handle vast literature volumes [59,61]. However, the latter is currently hindered by “hallucinations” and systematic scoring biases [60,61]. Consequently, a fully automated pipeline remains elusive, with the most successful models using automation for data-intensive retrieval and initial encoding [17,33], while retaining human oversight for contextual interpretation [60,65].

The findings of this review have direct implications for digital health developers, guideline bodies, and health informaticians. In the retrieval and extraction pillars, transformer-based classifiers and rule-based NLP tools are sufficiently mature for use as screening and extraction assistants in literature surveillance workflows, provided human oversight is retained for borderline cases and interpretive content [19,46,55].

In the encoding pillar, FHIR-based authoring tools—specifically the WHO DAK-to-FHIR pipeline, developed by Shivers et al [53] in collaboration with the WHO, and CPG-on-EBM-on-FHIR, developed by Lichtner et al [56] as part of the COVID-19 Evidence Ecosystem (CEOsys) project—are the outputs of this research program closest to practical deployment. They offer standardized, interoperable artifacts that can integrate with existing health information systems and support incremental updates. Neither has yet been evaluated in routine guideline production.

In the evaluation pillar, LLM-based tools show promise for ranking and flagging guideline quality but should not yet be used as stand-alone assessors given the documented upward scoring bias [26,61]; hybrid workflows combining LLM screening with human confirmation represent the most defensible current approach.

The most significant knowledge gap identified by this review is the absence of validated end-to-end systems connecting all 4 pillars: no existing study has demonstrated a complete pipeline from automated literature retrieval through extraction, encoding, and evaluation to recommendation generation at scale. This represents the primary development priority for the field.

Several features of the search constrain what this review can claim. The search covered 2 databases: PubMed and Scopus. Scopus was included to reach the informatics and engineering literature that a biomedical database indexes incompletely. Even so, it is likely that IEEE Xplore, the ACM Digital Library, and Web of Science would surface further work, particularly in conference proceedings. The search strategy is a building-block design that pairs guideline terms with named formalisms, tasks, and instruments; it favors precision over recall and may underretrieve studies describing comparable methods in vocabulary outside the enumerated pairings, and it was not formally peer-reviewed. Gray literature, preprints, and dissertations were not searched. Eligibility was restricted to English-language publications; one otherwise eligible study was excluded on this basis, and the restriction may underrepresent work published in other languages, particularly from European, Latin American, and East Asian informatics groups active in this field. The search window opens in January 2001, and earlier work is therefore not represented.

The synthesis of evidence was significantly challenged by the substantial study heterogeneity across the included literature. This variability encompassed differences in study design, target clinical domains, specific automation technologies evaluated, and reported outcomes, which collectively precluded the execution of a formal meta-analysis. Consequently, the findings presented are primarily based on a narrative and descriptive synthesis.

Furthermore, it is essential to contextualize the findings within the rapidly evolving landscape of the field, particularly concerning advancements in sophisticated language models such as LLMs and other forms of generative AI [4,61]. Given the pace of innovation, some of the specific technological findings, or the general technological state described in the older included literature (eg, pre-2015 NLP methods), may be potentially dated. Future research will need to continuously update this evidence base to remain current with emerging technologies.

A significant limitation observed across the corpus was that most studies were proof-of-concept demonstrations or early-stage feasibility assessments (Table 2). They often focused on showing what was technologically possible in a controlled environment. As a result, there was a limited body of evidence pertaining to real-world implementation, practical utility, and sustained impact. Crucially, evidence on long-term effectiveness, scalability, cost-effectiveness, and integration into existing clinical workflows remains scarce. The lack of robust implementation evidence makes it difficult to draw definitive conclusions about the true clinical and operational value of these automation solutions.

Finally, the potential for publication bias must be acknowledged. Research dissemination practices may inherently favor the reporting of studies that demonstrate positive or novel findings. This bias could lead to an overrepresentation of successful proof-of-concept evaluations, potentially skewing the perception of efficacy and feasibility, while studies with neutral or negative results, or those focused purely on implementation challenges, may be underrepresented in the published literature.

Automation of clinical practice guideline development has established technical feasibility across retrieval, extraction, encoding, and evaluation, with the strongest evidence for bounded, well-structured tasks assessed under controlled conditions. Evidence for end-to-end, production-scale living guideline pipelines remains limited: no study in this corpus demonstrated a validated pipeline connecting all 4 pillars in an operating guideline program, and evidence on clinical safety, workflow integration, and cost-effectiveness is largely absent. The pattern across the literature supports a hybrid, human-in-the-loop model in which automation handles data-intensive retrieval and initial encoding while experts provide contextual judgment and validation. Realizing continuously updated living guidelines will require both interoperability standards that connect extraction to computable representation, and evaluation of these components in real guideline production rather than in retrospective benchmarks.