Investigator
Duke Nus Medical School
Prompt Engineering for Eastern Cooperative Oncology Group Status Extraction: Comparing Large Language Model Techniques
PURPOSE Eastern Cooperative Oncology Group (ECOG) performance status is critical for cancer patient management, yet it is often documented only in unstructured clinical notes. This study compares several approaches to extract ECOG status from oncology notes, focusing on advanced prompting techniques for large language models (LLMs). METHODS We evaluated four ECOG extraction approaches on unstructured clinical notes from patients with non–small cell lung cancer, multiple myeloma, or ovarian cancer (2017-2021). The approaches were a rule-based natural language processing algorithm, simple LLM prompting, and two advanced prompts (chain-of-thought and Double Filtering) using a domain-tuned LLM (LLAMAv3.2). Performance was measured on a binary outcome (any ECOG documented v none) and a three-class outcome (ECOG 0-1 v ≥2 v none) and via an adapted QUEST questionnaire for human evaluation. RESULTS Both CoT and double filtering technique (DFT) achieved 94% accuracy, outperforming the rule-based method (91%) and simple prompting (86%). DFT had the highest specificity (0.91) and positive predictive value (PPV; 0.93), whereas CoT attained the highest sensitivity (0.98). In the QUEST evaluation, DFT and CoT scored higher on output quality, reasoning, bias reduction, and user satisfaction than the simple prompt. DFT received the top satisfaction rating. In the three-class analysis, DFT and CoT again performed best (accuracy 0.91 v 0.87) and DFT was most sensitive for ECOG ≥2 cases. Estimates for ECOG ≥2 remained imprecise because of the small sample (n = 20). All methods sometimes hallucinated ECOG status. CONCLUSION Advanced LLM prompting improved ECOG extraction over basic methods. DFT and CoT each showed specific strengths (DFT had higher PPV and user satisfaction; CoT achieved higher sensitivity). These approaches appear to be generalizable across cancer types. Key implementation considerations include computational cost and human oversight. Overall, advanced prompting can standardize ECOG documentation, accelerate patient cohort identification, and inform personalized treatment planning.
Leveraging Electronic Health Records to Examine the Real-World Rates of Cancer Genetics Referrals in a Singapore Health Care Cluster
PURPOSE Identifying patients with hereditary cancer syndromes through genetics referral enhances early detection and reduces healthcare costs. Despite potential benefits, genetics referral rates globally, including Singapore, remain low. This study investigates the real-world rates of genetics referrals in eligible cancer patients at Singapore’s largest healthcare cluster using Electronic Health Records. METHODS Referral criteria for genetics referrals were based on international guidelines. The institution’s data repository was queried for eligible patients with relevant diagnosis codes from 2017 to 2021. We assessed genetics clinic attendance among eligible patients to evaluate referral rates. Variations in referral rates over time were analysed using linear regression and two-tailed t -test. RESULTS Of the 10,080 patients eligible for a genetics referral, 17.1% (1719) were referred to a cancer genetics clinic. Breast, ovarian, colorectal, and endometrial cancers accounted for 42.9%, 33.5%, 11.3%, and 8.6% of referrals, respectively. Other tumour types accounted for 3.7% of referrals. Referral rates for suspected Hereditary Breast and Ovarian Cancer syndrome (HBOC)-related cancers were higher (19.4%) than referrals for suspected Lynch syndrome (11.9%). Among HBOC referrals, women (20.7%) were more likely to be referred than males (7.8%). From 2017 to 2021, we found an increase in referral rates for HBOC (12.8%-28.6%, P = .005) but not for Lynch syndrome-related indications (7.7%-13.5%, P = NS). The increase in referral rates for suspected HBOC in women was more significant than in men ( P = .03). CONCLUSION This study found lower referral rates for Lynch syndrome than HBOC, and identified a gender discrepancy, with men with HBOC being less likely to be referred. Efforts to increase referral rates should include raising clinician awareness and electronically identifying suspected cases, especially for male breast cancer and Lynch Syndrome.
