Assessing Suicide Risk Reduction in the Interval Before Death by Suicide Using Natural Language Processing

Original Source: Journal of Clinical Psychiatry, all rights reserved. —

There is enduring clinical apprehension around high-suicide-risk patients whose symptoms seem to quickly improve.¹ This worry centers on the possibility that what is perceived as improvement may be fleeting or, worse, could represent a means of evading treatment or an unspoken decision to die by suicide.^2,3 Although this pattern of risk reduction before suicide has long been appreciated clinically, few empirical investigations address this domain, probably due to this dynamic’s relative infrequency and difficulty tracking patients’ “authentic” intention.

Many studies, however, note that patients who were recently discharged from inpatient mental health treatment have a pronounced suicide risk burden.^4,5 Ruffino and colleagues suggest that patients who demonstrate the most improvement in inpatient mental health care are most likely to die by suicide when compared to other inpatient groups.⁶ While these patients’ changes are not necessarily feigned, their symptom improvements and associated risk reductions are typically time-limited. These transient improvement signals may lead clinicians, patient support networks, and patients themselves to mistakenly believe that risk has been decreased and that suicide prevention can, accordingly, be deprioritized.^2,6 Recognizing this finding, researchers have noted the difficulty of distinguishing between productive and dangerous improvement.³

In this study, we hope to gain understanding about patients who were deemed to be at high suicide risk and whose risk level appeared to decrease but who then proceeded to die by suicide. To better evaluate this population and the care that they receive, we leveraged prior studies’ insights, focusing on patients who were identified by their clinicians as being at a high suicide risk, who improved to the extent that their risk indication was removed and then died by suicide shortly thereafter. To better conceptualize this population, we contrast them with patients who similarly were selected by their clinicians as being at high suicide risk and who remained with that status until dying by suicide.

Our exploratory study has the joint objective of gaining epidemiological understanding of these patients by evaluating demographics and service utilization and investigating risk fluctuation in the interval leading up to their deaths. To achieve these goals, we include both structured and unstructured electronic health record (EHR) data, utilizing traditional statistical models and natural language processing (NLP) to maximize evaluation of clinical patterns and how they change over time.

METHODS

High-Risk Indication

The United States Department of Veterans Affairs (VA) is the largest integrated health system and mental health provider nationally.⁷ Veterans have a substantially elevated suicide risk burden when compared to their civilian counterparts.^8–10 As classifying suicide risk remains challenging,¹¹ beset by relatively low rates of suicide and diversity of symptom typologies, the VA has prioritized the development of targeted suicide risk prediction models.¹² These efforts focus primarily on identifying patients at the highest predicted suicide risk-tier, a subgroup that includes a disproportionate number of annual suicide deaths.^13,14

The VA uses a range of a suicide risk warning systems, including an alert system that identifies risk based on statistical modeling (REACH-VET) and a clinical high risk flagging (High Risk Flag [HRF]) system¹⁵ based on clinicians’ clinical concerns. HRF is initiated after documentation of suicidal behavior, including suicide attempts, pronounced suicidal ideation, and self-harm threats. HRF referrals are initiated by clinicians and then confirmed by facility suicide prevention coordinators, who work with the patients’ care team to address suicide risk and risk monitoring. Patients’ HRF status is entered into the EHR system, which alerts all clinicians about the flagged patient’s risk status upon opening the patient’s chart. HRF status is reevaluated approximately every 90 days and can be renewed indefinitely. Both REACH-VET and HRF are designed to be used as part of a stepped care approach, a clinical methodology that prioritizes more intensive care for those with elevated risk.¹⁶ Though related, REACH-VET and HRF operate separately within the VA suicide prevention program, each offering comparative strengths and weaknesses.^17,18 Within the context of this study, we utilized the HRF to capture clinical, rather than actuarial, dynamics.^19-22

Sample

We initially selected all VA patients who died by suicide and had at least 1 VA health care encounter from 2015 to 2018 (n=9,614). We developed this sample by linking cause of death data from the VA-Department of Defense Mortality Data Repository²³ with EHR data from the VA Corporate Data Warehouse, the central source for retrospective VA data. We restricted the sample to patients who had HRF status activity (n=938) in the year before death and then subselected those who had these statuses deactivated in this interval (n=250). We termed the interval in which patients had active HRF statuses as “Active-HRF,” the point of HRF removal as “HRF deactivation,” and the interval after HRF deactivation as “Post-HRF.” Finally, within this group, we identified patients who died by suicide within 180 days after HRF deactivation, a group we reference as Reduced Clinical Risk Classification (RCRC; n=121). We selected the 180-day duration as we wanted to track care differences and risk fluctuation during the Post-HRF interval. Our choice to select 180 days builds on related literature suggesting that VA patient suicide risk remains very elevated in the year after discharge from intensive psychiatric care,²⁴ but also that interventions targeting this interval can be particularly effective at preventing suicide.^5,25 While studies document that patients’ suicide-risk level was highest in the first 3 months following discharge, risk remained elevated for longer intervals.^25,26

To model comparisons between patients whose symptoms appeared to improve versus those whose symptoms remained consistently elevated, we also identified all patients who died by suicide while having Active-HRF statuses, a group we reference as Sustained Clinical Risk Classification (SCRC; n=138). As the SCRC patients had Active-HRF statuses until death, they did not have Post-HRF intervals. To determine the optimal caliper for matching on days between HRF activation and death, we plotted the number of 1: 1 matched RCRC and SCRC pairs across incrementally relaxed caliper widths (from exact match to ±40 days). The resulting curve exhibited a plateau beyond which further caliper relaxation yielded diminishing increases in matched pairs (Supplementary Figure 1). We selected the midpoint of this plateau as the final caliper width, representing the optimal balance between sample size and matching precision. We matched RCRC patients on a one-to-one basis with SCRC patients based on shared number of days (± 20 days) from HRF activation to death. Our final sample included 89 RCRC patients and 89 SCRC patients. Sample selection information is presented in Figure 1.

To evaluate comparisons during this interval, we based calculations on matched RCRC patient HRF deactivation dates. For example, if an RCRC patient had a Post-HRF interval of 30 days, we classified the 30 days prior to matched SCRC patient death as that SCRC patient’s Post-HRF interval. The duration of the groups’ respective Post-HRF intervals was comparable, as assessed by a Wilcoxon signed-rank test.²⁷ Relevant time points are clarified in Figure 2.

Structured Data

As demographics and clinical service usage variables are helpful in measuring suicide risk and discerning epidemiological patterns, we used resources from the VA’s Corporate Data Warehouse²⁸ to extract variables linked with suicidality,²⁹ including demographics, diagnoses, medication usage, service usage, and military exposure, and then calculated standardized mean differences (SMDs)³⁰ between the RCRC and SCRC groups. Additionally, we compared per diem rates of health care utilization during the Active-HRF and Post-HRF intervals for RCRC and matched SCRC patients. We selected these variables (number of inpatient mental health days and number of emergency department visits) as they are core predictors of suicide risk and are temporally labile. Following Andrade,³⁰ SMD values within the ranges of 0.2–0.49, 0.5–0.79, and 0.8–1 were interpreted as indicating, respectively, small, medium, and large differences between groups. We evaluated mean, standard deviation (SD), and interquartile range (IQR), a measure of statistical dispersion based on the difference between the data’s third quartile (Q3) and the first quartile (Q1). We also calculated Wilcoxon signed rank test²⁷ to derive P values and measure statistical significance for the structured variables.

Unstructured Data

The analysis of unstructured EHR documents using NLP is increasingly used to derive nuanced psychosocial patterns and monitor treatment dynamics in suicide prediction research.^31,32 Latent Dirichlet Allocation (LDA) is a widely used NLP method that derives summary topics in text corpora. LDA works under the assumption that each text document in a corpus comprises words drawn from a mixture of underlying topics. The word occurrences in each document are modeled as multinomial distributions.³³ Each topic is characterized by a Dirichlet distribution, which assigns probabilities to words based on their relevance to that topic—these probabilities sum to 1, reflecting the topic’s internal composition. The distribution of topics in each document can be modeled using a Dirichlet prior, capturing the relative prevalence of a given topic. LDA follows a generative approach in which, for each document, topics are sampled from the document-specific topic distribution, and then words are sampled from the topic-specific word distribution. In this case, to prevent patients with more documents from having more influence over the topics than patients with fewer documents, the word frequencies for each document were adjusted to account for the number of notes for each patient. This weighting method allows each patient to contribute equally to the resulting topics.

Following this approach, we extracted all text integration utilities documents (clinical notes) from all administratively coded VA encounters from HRF indication until death for RCRC and SCRC patients (20,791 and 40,844 documents, respectively). We then performed tokenization, a process that splits text data into units of 1 or 2 words or punctuation marks called “tokens,” removed common words (“stop-words”) that were premised to offer little added meaning to our model (words like “and” or “the”), as well as terms that were ubiquitous and uninformative in our data set (like “vet” or “veteran”), eliminated symbols and punctuation, and made all words lowercase. We then ran a weighted LDA on the RCRC group’s Active-HRF corpus and derived relevant summary topics. We selected this corpus to specifically model care differences and improvement indicators. To determine the optimal number of topics, we used the “knee point” method on the coherence curve, examining how coherence changed with varying topic numbers (Supplementary Figure 2). This method determined that increasing the number of topics after 6 did not substantially improve coherence. However, the intertopic distance plot for a 6-topic model showed significant overlap of 2 topics defined by medication-related terms. These 2 topics can be seen in Supplementary Figures 3 and 4. When we ran a 5-topic model, the medication-related topics merged, and the intertopic distance plot showed distinct, nonoverlapping topics (Supplementary Figure 5), indicating semantic separation of topics. We moved forward with the 5-topic model for this reason.

The topic-document distributions were obtained for all documents in the corpus by applying the model to the unweighted word frequencies from all documents in both RCRC and SCRC Active-HRF and Post-HRF intervals. These topic-document distributions describe how much of each document is described by each topic as a percent, such that the total sums to 1. As such, all documents were aligned with at least 1 topic. By training on the weighted corpus and applying this trained model to the entire unweighted corpus, we established patient-level topics while avoiding topic shift when extracting document-level topic distributions.

We evaluated topic drift and prevalence of random distributions of these topics across both RCRC and SCRC groups by visual inspection of the histogram of maximum topic difference (defined as maximum topic proportion minus minimum topic proportion) for all documents, stratified by clinical risk classification (Supplementary Figure 6). A document with random topic probabilities for all topics would have a maximum topic difference of 0%. This was observed in less than 1% of documents of either clinical risk classification. The median maximum topic difference was 0.80 (IQR: 0.61, 0.96) for SCRC documents and 0.83 (IQR: 0.64, 0.96) for RCRC documents. The plots also display similarly shaped distributions, indicating topic stability.

For each topic, we employed a generalized additive mixed model (GAMM)³⁴ to predict how the topic prevalence changes across time and risk classification status, accounting for repeated measures within individual patients with a random effect. This random effect of patients also accounted for variable number of documents and time between HRF activation and death contributed by each patient. We chose to use GAMMs because we suspected the effect of time on topic proportion would be nonlinear. Using 1 GAMM model per topic also gave us more power to discern topic differences over time than a binned approach since all data could be evaluated simultaneously. The degree of smoothness was estimated using restricted maximum likelihood,³⁵ a modeling approach that enabled us to estimate predicted topic proportion over time and evaluate statistical differences for topic presence between groups. This approach is noted for reducing overfitting concerns within GAMM analysis.^35,36 Full model specifications are presented in Supplementary Table 1. Visualization of topic-word relevance was carried out using pyLDAvis.³⁷

Finally, a multidisciplinary clinical panel (psychiatrist, social worker, and psychologist) independently reviewed topic content and proposed initial labels. We assessed concordance qualitatively based on initial overlap in proposed labels and addressed discrepancies through a structured consensus process conducted over 3 rounds of discussion, during which reviewers compared interpretations, refined label definitions, and resolved disagreements. Final topic labels were assigned once unanimous agreement was reached across panel members.

We extracted estimated marginal means from our GAMMs using the emmeans package.³⁸ This extraction procedure is detailed in Supplementary Table 1. Predicted topic proportions and confidence intervals were plotted over time for SCRC and RCRC patients. These analyses allowed us to compare treatment patterns during RCRC patients’ Active-HRF and Post-HRF intervals and, broadly, to compare the treatment patterns of RCRC and SCRC patients in the intervals leading to their deaths. Nonoverlapping confidence intervals were interpreted as indicative of statistical differences.³⁹ As our analyses were exploratory, we did not conduct formal hypothesis tests nor apply multiple-comparison corrections.

The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Declaration of Helsinki of 1975, as revised in 2013. All procedures involving human subjects/patients were approved by the Veterans Institutional Review Board of Northern New England. This study was completed as part of the Leveraging Natural Language Processing to Evaluate Suicide Risk in Electronic Medical Record Notes project (1582333). Our review board determined that informed consent was not needed under the 2018 Common Rule, given the study’s reliance on retrospective EHR data. We used Gensim⁴⁰ and Scikit-learn⁴¹ to complete LDA modeling in Python. Additional analyses were completed in R⁴² using the base R Stats for modeling⁴³ and emmeans³⁸ to extract predicted topic probabilities across time. This study was not preregistered. Data access is restricted due to the clinical nature of dataset and VA privacy protections.

RESULTS

Sample

Our final sample consisted of 89 matched RCRC and SCRC patients. The sample’s demographics and health care utilization are described in Table 1. RCRC patients tended to be somewhat younger and have less disability and fewer mental health diagnoses than SCRC patients.

Structured Data

SCRC patients had significantly and nearly significantly more inpatient mental health days per day during the Active-HRF and Post-HRF intervals, respectively, than RCRC patients. SCRC patients also had significantly more emergency department visits per day during the Active-HRF interval. Patient rates of emergency department visits per day during the Post-HRF interval, however, were similar. For both groups, the average number of encounters substantially declined from Active-HRF to the Post-HRF intervals. Full structured data comparisons are presented in Table 2.

Unstructured Data: RCRC Topics

Our initial analysis of the RCRC Active-HRF corpus yielded 5 summary topics: Psychotherapy, characterized by words like “treatment,” “session,” “discussed,” and “depression”; Medication, characterized by words like “medication,” “tablet,” “milligram,” and “refill”; Outpatient, characterized by words like “assessment,” “education,” “intervention,” and “procedure”; Follow-up, characterized by words like “call,” “appointment,” “contact,” and “phone”; and Suicide Assessment, characterized by words like “suicide,” “risk,” “safety,” and “flag”.

Unstructured Data: Fluctuations

To gauge topic fluctuations over time for the RCRC and SCRC groups, we evaluated topic probabilities from HRF activation until patient death based on the RCRC-trained model. This analysis showcased significantly different topic probability patterns for the RCRC and SCRC groups, as presented in Figure 3. For all topics, RCRC and SCRC probabilities were similar during the initial Active-HRF interval. Contrasting topic patterns, however, were identified during the subsequent care interval, with differences becoming starker after HRF deactivation.

Within the Psychotherapy topic, while SCRC patients’ probabilities remained consistent, RCRC patients’ probabilities were bell-shaped, declining over the course of flagging, before peaking just before HRF deactivation and then declining again in the deactivation interval. Within both the Medication and Outpatient topics, group topic probability patterns were similar until after HRF deactivation, after which RCRC patients’ probabilities substantially increased, while SCRC patients’ probabilities, respectively, remained level or substantially decreased. Within both the Follow-up and Suicide Assessment topics, group topic probability patterns were similar until after HRF deactivation, after which RCRC patients’ probabilities substantially decreased, while SCRC patients’ probabilities substantially increased. For RCRC and SCRC patients who lived for a longer duration after HRF deactivation, topic probability differences became more pronounced, except for the Psychotherapy and Medication topics, in which probabilities tended to converge. Overall, RCRC patients showed greater probability fluctuation during the active period when compared to SCRC patients. During Post-HRF, RCRC patients experienced marked declines across multiple care domains, particularly psychotherapy, outpatient visits, follow-up, and suicide-specific assessments, while SCRC patients, who remained with HRF active statuses, saw sustained or increased engagement within these domains.

DISCUSSION

This exploratory and hypothesis-generating study focused on gaining understanding about RCRC patients, VA patients who were classified as being at high suicide risk, who over the course of treatment appeared to improve, such that they were removed from intensive risk monitoring, but who nonetheless proceeded to die by suicide within the following 6 months. To contextualize the unique qualities of this group and the ways treatment impacted them, we compared them with SCRC patients, patients who were similarly classified as high-risk, but whose risk level remained elevated and died by suicide. Rather than leading with specific hypotheses, the study was data-driven, allowing initial exploration of these patient populations’ clinical patterns.

We identified small differences in these groups’ demographic and health care utilization patterns, with SCRC patients tending to have higher levels of symptoms and utilize more clinical services than RCRC patients. These differences were apparent in general and, specifically, during the Active-HRF interval. These findings suggest that RCRC patients’ comparatively lower levels of illness may have informed their clinical care, including their clinicians’ decision to remove their HRF status, allowing prioritization of patients with perceived higher symptom burden and acute suicide risk.

This dynamic was similarly apparent in our topic-modeling analysis, which identified 2 primary topic patterns, both of which captured the RCRC group’s increased fluctuation relative to the SCRC group. The first pattern, that of the Psychotherapy topic, showed substantial fluctuations over the course of the Active-HRF interval, with a substantial rise in psychotherapy-related content in the lead-up to HRF deactivation, followed by a marked decline immediately after deactivation. The second pattern, that of all other topics, exhibited minor fluctuations during the Active-HRF interval followed by substantial changes after deactivation.

The Psychotherapy topic showcased the respective groups’ unique care trajectories. Although initially RCRC patients had significantly lower Psychotherapy topic representation than SCRC patients, over the course of the Active-HRF interval, this dynamic flipped, such that immediately before deactivation RCRC patients had significantly higher Psychotherapy topic representation than SCRC patients. After deactivation and the accompanying shift toward less intensive clinical care and risk safeguards, RCRC patients’ psychotherapy topic representation dropped markedly, such that they again had significantly lower Psychotherapy topic representation than SCRC patients. While the increase in Psychotherapy topic representation during Active-HRF is likely associated with perceived symptom reduction, the postdeactivation decline suggests perceived symptom increase.

This Psychotherapy topic fluctuation infers a range of clinical dynamics that could impact care trajectories. It is likely that, as RCRC patients had lower perceived symptom burdens, clinicians may utilize treatments designed for patients with less acute risk.^44,45 It is also possible that clinicians felt less concern about working with patients who they perceived to have less acute suicide risk,⁴⁶ which, in turn, allowed for improved therapeutic alliance and countertransference with these patients.⁴⁷ On a more subtle level, the RCRC topic fluctuations during the Active-HRF interval may highlight a rupture-repair alliance pattern, a dynamic in which initial strains lead to deeper relationship bonds.⁴⁸ This is contrasted with the SCRC trajectory, which remained flat throughout the treatment interval and could reflect a static or comparatively less developed alliance pattern.^46,48

In contrast to the Psychotherapy topic, where differences between the patient groups were apparent both during and after HRF declassification, for the other topics, differences only became apparent after declassification. This divergence suggests that these topics relate to differences associated with RCRC patients receiving less intensive suicide prevention care.

Whereas SCRC patient topic probabilities remained constant during the Post-HRF period, except for in the Follow-up topic, in which probabilities surged, and the Outpatient topic, in which probabilities continued a downward trajectory, the topic patterns intimate that RCRC patients experienced a marked reduction in clinical contact and supervision after HRF deactivation. While the dramatic increase in Medication content may reference a spike, or at least a refocus, on clinical needs after HRF deactivation, it could also reflect RCRC patients’ increased utilization of home-based medications and decreased usage of intensive inpatient services. This differential captures SCRC patients’ continued intensive risk monitoring and sustained inpatient care utilization in juxtaposition to RCRC patients’ shift toward less intensive care contexts. These trends affirm previous research about patients perceived to be at lower risk receiving less intensive mental health resources.^13,49

Accentuating prior findings about the pronounced suicide risk following completion of inpatient mental health interventions,^4,50,51 it is likely that RCRC patients, upon HRF deactivation, encountered compounded risk vulnerabilities. Withdrawal from intensive risk monitoring and associated care likely reduces opportunities to maintain prevention strategies and preemptively identify crises. It could also contribute to a sense of thwarted belongingness and perceived burdensomeness, risk factors conceptualized within the interpersonal theory of suicide.⁵²

Implications

Our study hoped to gain understanding about VA patients who experienced substantial suicide risk reduction before dying by suicide. Rather than detecting feigned symptom change, our exploratory findings suggest a pattern of perceived symptom improvement that may have contributed to reduced surveillance, fewer follow-up contacts, and diminished suicide-specific monitoring in the interval leading up to patients’ death by suicide. It makes sense that without the additional surveillance and treatment that HRF patients receive, suicide risk could quickly resurface, regardless of improvement authenticity, in the Post-HRF interval. Indeed, as has been widely cited about the interval immediately after inpatient mental health visits,⁵⁰ it is likely that the period after HRF deactivation is a time of particularly elevated suicide risk.

It is of consequence that RCRC patients appear to have lower levels of pathology and symptoms to begin with than SCRC patients. This may have the dual function of allowing RCRC patients greater capacity for personal behavioral change and enhanced relationships with their clinicians when compared to patients whose risk level remains consistently more elevated.^46,53 This combination positions RCRC patients in a context that could allow them to evade risk detection. This pattern intimates that clinicians, consciously or unconsciously, may be less accurate in measuring and monitoring the suicide risk of patients with perceived lower symptom burdens, when compared to patients with perceived higher symptom burdens.

It is important to note that these clinicians, and health systems more broadly, frequently lack resources to reach all patients at risk for suicide.^54,55 Given this demand for resources, clinicians are often encouraged to transition patients with lower risk to less resource-intensive care frameworks to ensure that patients with higher classified suicide risk are prioritized. Furthermore, following literature about psychotherapeutic alliance, countertransference, and suicide risk^46,56 and our prior NLP analysis of lexical patterns in medical corpora,^57–59 it is possible that, when comparing these patient groups and their relationships with their clinicians, clinicians may have utilized divergent care formats with these respective patient populations. Rather than suggesting that this is an active choice, or an act of manipulation, our findings suggest this may stem from perceptions that RCRC patients were at lower suicide risk and potentially that clinicians had stronger alliances with these patients.^45,46

As broadly discussed in the literature,²⁹ suicide risk prediction, including clinical risk detection and algorithm risk detection methods, remains imprecise. To better capture suicide risk, recent research suggests the importance of leveraging diverse data formats, including psychosocial risk variables derived from unstructured EHR and time-sensitive risk variables.^58,60 Improving risk detection could help identify patients’ actual risk level and ensure added safety protections. Our exploratory findings suggest that clinicians may overly rely on active symptom assessment to classify risk, but that utilizing more expansive methods could lead to more comprehensive and accurate risk appraisals. These results underscore the need for multidimensional reappraisal strategies that account for both symptomatic improvement and contextual risk factors, including social disconnection, recent transitions, and fluctuations in treatment adherence. To mitigate associated risks for patients who respond positively to treatment, authentically or not, our research suggests the importance of advancing new standards in risk evaluation, treatment resources, clinician self-awareness, and care transitions.

Current VA prevention methods, which center on intensive care for the highest-risk population, unfortunately leave patient populations with lower, but still elevated, risk substantially underserved.¹⁴ Following a public health approach to suicide prevention,^61,62 we argue in favor of suicide-risk-tier–specific interventions and the development of less intensive suicide prevention interventions and resources. Lastly, we suggest that following HRF deactivation, or other such indicators of suicide risk improvement, there should be a transition framework to ease the care de-escalation process.⁶³ Alongside other processes that have been successfully developed to scaffold related care transitions and are now being introduced in other aspects of VA mental health care,⁵ our findings indicate the potential value of a safety framework to support patients transitioning off from HRF.

Limitations

It is important to reaffirm that our study utilized an exploratory topic modeling method⁶⁴ rather than a hypothesis-driven inference method⁶⁵ and that our identified topics were based on textual patterns rather than direct measures of clinical processes. Our findings, accordingly, need further validation using independent clinical measures. Moreover, our study was constrained by a range of limitations, including inclusion criteria, sample size, matching, and computational resources. We conceptualized high risk status based on HRF indication and risk reduction based on removal from HRF indication. Alternative risk assessment formats, such as a REACH-VET score, an algorithmic risk metric, or the Columbia-Suicide Severity Rating Scale, a self-report metric, could have led to a very different sample.⁶⁶ Specifically, as inclusion criteria required patients to be diagnosed by a clinician, we may have selected patients with greater likelihoods to be somewhat engaged in care. Our analytic ability was somewhat hindered by small sample sizes. It is possible that, even though our sample is closely matched, analyses may contain artifacts of sample size constraints. Lastly, although more advanced topic modeling techniques such as BERTopic could support our findings or offer improvements, the computational and software restrictions of the server on which we conducted these analyses meant that these techniques were unavailable to us.

Our dataset may not be generalizable to broader VA populations due to our sample’s higher levels of older, female, and racial minority Veterans.⁶⁷ As roughly 35% of Veterans receive mental health care outside of the VA, it is likely that some patients in our sample received additional psychotherapy sessions outside of VA contexts.⁶⁸ Unfortunately, evaluating non-VA data is not feasible within this study. As findings were exclusively based on a specific VA patient population, they should not be generalized to civilian populations and should be interpreted with caution regarding other VA patient populations. Similarly, as our analyses were only based on VA clinicians, our findings should not be generalized to clinicians outside the VA. Given data limitations, it was not feasible to control for specific intervention formats. During the study window, suicide risk screeners were not widely utilized within the VA; hence, we were not able to include these data in our analysis. In future studies, where we have access to a larger and more contemporary dataset, we hope to compare RCRC patients who died and did not die by suicide. As both of our comparison groups died by suicide, our study does not aid identification of treatment patterns that prevent suicide. We are optimistic that upgraded computational abilities, VA EHR advancements, and access to newer data will enable enhanced measurement of changes over time and specific intervention utilization.

It is important to emphasize that our study utilized retrospective data. While these data allowed us to track outcomes and mortality trends, we do not have the ability to truly ascertain patients’ clinical presentations or the nuance of patients’ and clinicians’ interactions. The clinical meaning of our identified topics and the lexical differences between patient populations require additional analysis and validation. While our access to unstructured EHR data allows some ability to evaluate clinical dynamics, we are cautious about interpreting lexical frequency as indication of care quality. We hope that our study can lead to more clinical attention on patients whose perceived suicide risk burden appears to quickly improve. While our analysis was based on a small patient population who represent a limited percentage of VA suicide decedents, the detected dialectical pattern of heightened care, symptom reduction, diminished oversight, and death by suicide has relevance across the VA’s various suicide prevention mechanisms.⁵⁵

Our study aids characterization of VA patient populations who have historically been poorly understood, highlighting their demographics, service usage, and treatment characteristics. Future studies could use this information to refine predictive modeling and risk-tier-specific intervention methods. Subsequent research could use this approach to assess temporal topic fluctuations and associations with risk and prevention, including evaluations of psychotherapeutic alliance and treatment engagement, core predictors of outcome.^69,70

Article Information

Published Online: October 16, 2026. https://doi.org/10.4088/JCP.25m16249
© 2026 Physicians Postgraduate Press, Inc.
Submitted: December 1, 2025; accepted August 18, 2026.
To Cite: Levis M, Dimambro M, Levy J, et al. Assessing suicide risk reduction in the interval before death by suicide using natural language processing. J Clin Psychiatry 2026;87(4):25m16249.
Author Affiliations: VAMC White River Junction, White River Junction, Vermont (Levis, Dimambro, Watts, Shiner); Department of Psychiatry, Geisel School of Medicine, Hanover, New Hampshire (Levis, Watts, Shiner); Departments of Pathology and Laboratory Medicine, Cedars-Sinai, Los Angeles, California (Levy).
Corresponding Author: Maxwell Levis, PhD, White River Junction VA Medical Center, 163 Veterans Dr, White River Junction, VT 05009 ([email protected]).
Financial Disclosure: The authors have no conflicts of interest.
Funding/Support: This work was funded by Dr Levis’ VA Clinical Science Research and Development Career Development Award (1IK2CX002630-01A1).
Data Availability Statement: This study exclusively used VA data and followed VA data human studies protocols. Data access is restricted due to the clinical nature of the dataset and VA privacy protections.
Supplementary Material: Available at Psychiatrist.com.

Clinical Points

• Assessing and treating suicide risk is complex, especially for high-suicide-risk patients who quickly respond to psychotherapeutic treatment.
• This study suggests the importance of monitoring these patients’ suicide risk even after they are no longer classified as being at high suicide risk.

Journal of Clinical Psychiatry, all rights reserved.