Title of Research Proposal: Predicting Treatment Efficacy in Individuals with Major Depression — Deep Neural Networks for Physiological Patient Parameters
Potential Conflicts of Interest: DB, CA, RF, KP, SI are officers or shareholders in Aifred Health, the mental health AI startup collaborating with the Douglas on this project. Conflicts of interest are managed as all publications must pass through Dr. Turecki, who has no interest in the company, as must the direction of the research project. All conflicts have also been declared as part of the successful local REB application..
Major Depression is a serious mental illness that globally affects 11.1% of people over the course of their lives and that is projected to be responsible for the majority of Disability-Adjusted Life Years (DALY’s) lost by 2030. While a range of effective treatments do exist, these are not equivalently effective for all patients and some patients can spend years finding the right choice from the dozens of medications, multiple psychotherapies, and five neurostimulation techniques available.
Currently, most patients and their physicians have little option but to go through a “guess and check” approach to finding the right treatment. For a patient with depression, trying a new treatment means several weeks of therapy or medication titration to start seeing if there is a positive effect. This is time lost in the patient’s life — time that is potentially away from work and when they are not able to be fully present in their families’ lives. Inadequately treated depression also leads to risks of suicide and self-harm. What’s more, many patients with depression will not improve after the first treatment — in the STAR*D trial, only about one third of patients improved after their first treatment trial, with decreasing response rates after further trials. This means that the decision about which treatment to try is one that has significant consequences.
It is clear that a research objective for depression, other than improving diagnostic rates and access to care, should be developing an evidence-based approach for rapidly selecting the most effective treatment for a given patient, as early on in their clinical course as possible, while minimizing side effects that lead to reduced quality of life or treatment adherence. Existing psychiatric guidelines do separate the large of array of treatment options into first, second, and third line treatments; and clinical experience has taught mental health professionals that certain types of medications or psychotherapy approaches work best in certain kinds of patients. Different patients develop different side effects to the same medication in an often unpredictable manner, further complicating treatment choice. However, there is not a systematic, evidence-based tool that predicts treatments in a way that is personalized to a given patient.
Statistical Analysis Plan:
Data will be stored as per data privacy regulations in Canada and as highlighted by each country’s Office of Privacy Commissioner or equivalent institution (as we will be receiving data from the US and the United Kingdom). We will comply with offshore data storage requirements as well as each country’s healthcare data privacy regulations. A secure environment will be set up, accessible by Secure Shell (SSH) keys. Internal to the environment, all data are to be encrypted by using current and strong cipher suites as identified by the National Institute of Standards and Technology. Role-based access control and Identify and Access Management will be set up to manage users with limited admin profiles. Standard hardening procedures will be applied on the node (Linux). A hardware security module will be used to manage the generation of SSH keys. Keys will be rotated through a key management system as recommended by reference security organizations. Audit logs will be generated for forensics and non-repudiation. Finally, we will use a Security Information and Event Management system in conjunction with the above.
With respect to statistics, as noted above, sensitivity, specificity, Positive Predictive Value, Negative Predictive Value, and Area Under the Receiver Operating Curve (AUROC) will be used to test each iteration of the AI system. These evaluation metrics have been chosen so that we can easily communicate our results with medical professionals as to how well the algorithms are performing. The ROC curve will help us be more conservative when predicting that a specific treatment will not work, especially in the case of more extreme measures.
Requested Studies:
A MULTICENTER, RANDOMIZED, DOUBLE-BLIND, PLACEBO-CONTROLLED, PARALLEL-GROUP STUDY TO EVALUATE THE EFFICACY AND SAFETY OF 2 FIXED DOSES (10 MG AND 50 MG/DAY) OF DVS SR TABLETS IN ADULT OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER.
Data Contributor: Pfizer Inc.
Study ID: NCT00863798/B2061005
A DOUBLE-BLIND, RANDOMIZED, PLACEBO-CONTROLLED TRIAL OF PRISTIQ IN FUNCTIONALLY IMPAIRED PATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT00824291/B2061006
A MULTICENTER, RANDOMIZED, 8-WEEK DOUBLE-BLIND ACUTE PHASE FOLLOWED BY A 6-MONTH CONTINUATION PHASE (OPEN-LABEL OR DOUBLE-BLIND) STUDY TO EVALUATE THE EFFICACY, SAFETY AND TOLERABILITY OF DVS SR VERSUS ESCITALOPRAM IN POSTMENOPAUSAL WOMEN WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT00406640/B2061096
AN 8-WEEK STUDY OF DVS VS. PLACEBO IN PERI- AND POST-MENOPAUSAL WOMEN (>40 YEARS)
Data Contributor: Pfizer Inc.
Study ID: NCT00369343/B2061097
A Multicenter, Randomized, Double-Blind, Placebo-Controlled, Parallel Group Study To Evaluate The Efficacy And Safety Of DVS-233 SR For Prevention Of Depressive Relapse In Adult Outpatients With Major Depressive Disorder
Data Contributor: Pfizer Inc.
Study ID: NCT00075257/B2061068
A 10 MONTH OPEN-LABEL EVALUATION OF THE LONG-TERM SAFETY OF DVS-233 SR IN OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT01309542/B2061069
A Multicenter, Randomized, Double-blind, Placebo-Controlled, Parallel-Group, Efficacy and Safety Study of a Flexible Dose of DVS-233 SR in Adult Outpatients With Major Depressive Disorder
Data Contributor: Pfizer Inc.
Study ID: NCT00063206/B2061070
MULTICENTER, RANDOMIZED DOUBLE-BLIND, PLACEBO-CONTROLLED STUDY OF THREE FIXED DOSES OF DVS-233 SR IN ADULT OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT00072774/B2061072
RANDOMIZED, DOUBLE-BLIND, PLACEBO CONTROLLED STUDY OF DVS-233 SR IN INPATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT00073762/B2061074
MULTICENTER, RANDOMIZED, DOUBLE-BLIND, PLACEBO AND COMPARATOR CONTROLLED STUDY OF A FLEXIBLE DOSE OF DVS-233 SR IN OUTPATIENTS WITH MAJOR DEPRESSION
Data Contributor: Pfizer Inc.
Study ID: NCT00090649/B2061075
MULTICENTER, RANDOMIZED, DOUBLE-BLIND, PLACEBO-CONTROLLED STUDY OF A FIXED DOSE OF DVS-233 SR IN OUTPATIENTS WITH MAJOR DEPRESSION.
Data Contributor: Pfizer Inc.
Study ID: NCT00087737/B2061081
A MULTICENTER, RANDOMIZED, DOUBLE-BLIND, PLACEBO-CONTROLLED, PARALLEL-GROUP, EFFICACY AND SAFETY STUDY OF DVS-233 SR IN OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER.
Data Contributor: Pfizer Inc.
Study ID: NCT00092911/B2061083
A 12-MONTH, OPEN-LABEL, EVALUATION OF THE LONG-TERM SAFETY OF DVS-233 SR IN OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT00452595/B2061082
A MULTICENTER, DOUBLE-BLIND, PLACEBO-CONTROLLED, RANDOMIZED WITHDRAWAL, PARALLEL GROUP STUDY TO EVALUATE THE EFFICACY AND SAFETY OF 50 MG/DAY OF DVS SR IN ADULT OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT00887224/B2061004
A PHASE IV, MULTICENTER, RANDOMIZED, 8-WEEK, DOUBLE-BLIND, PLACEBO-CONTROLLED, PARALLEL-GROUP STUDY TO EVALUATE THE EFFICACY OF 2 FIXED DOSES (50 AND 100 MG/DAY) OF DESVENLAFAXINE SUCCINATE SUSTAINED-RELEASE FORMULATION (DVS SR) IN ADULT OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT01432457/B2061028
A MULITCENTER, PARALLEL-GROUP, RANDOMIZED, 10-WEEK, DOUBLE-BLIND, PLACEBO-CONTROLLED STUDY TO EVALUATE THE EFFICACY AND SAFETY OF 50 MG OF DESVENLAFAXINE SUCCINATE SUSTAINED-RELEASE FORMULATION (DVS SR) IN THE TREATMENT OF PERI- AND POST-MENOPAUSAL WOMEN WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT01121484/B2061029
A MULTICENTER, RANDOMIZED, DOUBLE-BLIND, PLACEBO-CONTROLLED, FLUOXETINE-REFERENCED, PARALLEL-GROUP STUDY TO EVALUATE THE EFFICACY, SAFETY AND TOLERABILITY OF DESVENLAFAXINE SUCCINATE SUSTAINED RELEASE (DVS SR) IN THE TREATMENT OF CHILDREN AND ADOLESCENT OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT01372150/B2061014
A MULTICENTER, RANDOMIZED, DOUBLE-BLIND, PLACEBO-CONTROLLED, PARALLEL-GROUP STUDY TO EVALUATE THE EFFICACY, SAFETY AND TOLERABILITY OF DESVENLAFAXINE SUCCINATE SUSTAINED-RELEASE (DVS SR) IN THE TREATMENT OF CHILDREN AND ADOLESCENT OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT01371734/B2061032
A MULTICENTER, RANDOMIZED, DOUBLE-BLIND, PLACEBO-CONTROLLED, PARALLEL GROUP STUDY TO EVALUATE THE EFFICACY AND SAFETY OF TWO FIXED DOSES (25 MG AND 50 MG/DAY) OF DVS SR IN ADULT OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT00798707/B2061003
A 10-MONTHS OPEN-LABEL EVALUATION OF THE LONG-TERM SAFETY OF DVS-233 SR IN JAPANESE ADULTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT00831415/B2061002
A 6-MONTH, OPEN-LABEL, MULTI-CENTER, FLEXIBLE DOSE EXTENSION STUDY TO THE B2061032 STUDY TO EVALUATE THE SAFETY, TOLERABILITY AND EFFICACY OF DESVENLAFAXINE SUCCINATE SUSTAINED RELEASE (DVS SR) TABLETS IN THE TREATMENT OF CHILDREN AND ADOLESCENT OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT01371708/B2061030
A 6 MONTH, OPEN LABEL, MULTI CENTER, FLEXIBLE DOSE EXTENSION STUDY TO THE B2061014 STUDY TO EVALUATE THE SAFETY, TOLERABILITY AND EFFICACY OF DESVENLAFAXINE SUCCINATE SUSTAINED RELEASE (DVS SR) TABLETS IN THE TREATMENT OF CHILDREN AND ADOLESCENT OUTPATIENTS WITH MAJOR DEPRESSIVE DISORDER
Data Contributor: Pfizer Inc.
Study ID: NCT01371721/B2061031
A RANDOMIZED, DOUBLE-BLIND PARALLEL GROUP STUDY TO COMPARE DISCONTINUATION SYMPTOMS IN ABRUPT DISCONTINUATION VERSUS A 1-WEEK TAPERING REGIMEN IN SUBJECTS WITH MAJOR DEPRESSIVE DISORDER (MDD) TREATED FOR 24 WEEKS WITH OPEN-LABEL 50 MG DESVENLAFAXINE SUCCINATE SUSTAINED-RELEASE FORMULATION (DVS SR)
Data Contributor: Pfizer Inc.
Study ID: NCT01056289/B2061010
MULTICENTER, RANDOMIZED, DOUBLE-BLIND, PLACEBO-CONTROLLED, PARALLEL GROUP STUDYTO EVALUATE THE EFFICACY AND SAFETY OF TWO FIXED DOSES (50 MG, 100 MG) OF DVS-233 SR IN ADULT OUTPATIENTS WITH MAJOR DEPRESSION
Data Contributor: Pfizer Inc.
Study ID: NCT00277823/B2061084
MULTICENTER, RANDOMIZED, DOUBLE-BLIND, PLACEBO-CONTROLLED, PARALLEL GROUP STUDY TOEVALUATE THE EFFICACY AND SAFETY OF TWO FIXED DOSES (50 MG, 100 MG) OFDVS-233 IN ADULT OUTPATIENTS WITH MAJOR DEPRESSION
Data Contributor: Pfizer Inc.
Study ID: NCT00300378/B2061085
STUDY OF 3 FIXED DOSES OF DVS SR IN MDD
Data Contributor: Pfizer Inc.
Study ID: NCT00384033/B2061041
Summary of Results:
The results of this data request were not strong enough to merit academic publication. The results of the analysis were as follows:
This is a model of depression treatment outcomes for a single drug, desvenlafaxine. 2,607 patients were sampled, stratified by the remission variable into a split of 2,111 in the training group and 235 in the held-out test group. We used the SGD optimizer and a single layer model architecture with a dropout value of 0.5 from the open source Vulcan project [https://github.com/Aifred-Health/VulcanAI]. This was a non-optimized model aimed at probing differences in performance introduced by adding data from later in the treatment cycle. As this was a toy model aimed at probing mostly the impact of adding the treatment variables from later in treatment, baseline features were selected based on completeness (i.e. features with the least missingness, rather than those best at predicting the outcome; features with the least missingness did not, unfortunately, line up with features we have previously found to be predictive at baseline). In the first model, we predict outcome (remission, binary) with baseline variables; in the second model we predict outcome with baseline variables with the addition of the second visit depression severity score. Model features included in each model are provided in a list below
baseline model: remission, anhedonia_general, psychomotor_retardation, sex_Female, Sadness, somatic_gastro_intestinal, depression severity score, sex_Male, age, race_Asian, race_Black, race_Hispanic or Latino, race_White, race_Other
trajectory model: remission, anhedonia_general, psychomotor_retardation, sex_Female, Sadness, somatic_gastro_intestinal, depression severity score, sex_Male, age, race_Asian, race_Black, race_Hispanic or Latino, race_White, race_Other, visit two depression severity score
As can be seen, adding data from later in the treatment cycle did not improve outcome prediction, as opposed to what we hypothesized and what has been demonstrated in other work, but overall model performance is poor, suggesting the need for improved optimization with a neural network architecture and rigorous checks of data quality. The former may be provided by Bayesian optimization techniques, which we have used in previous work and which has yielded improved results. Imputation techniques would also make more data available; they were not used here purely because of the focus on adding in post-baseline variables and focusing on their impact. This will be trialed in future analyses using multiple drugs. The overall poor results here essentially demonstrate that to leverage a neural network properly in these kinds datasets, we cannot rely on single data points but must properly optimize the model over the entire dataset. This is intended as a teaching point for further work; the results themselves are not intended to have clinical meaning. Future models will help use determine if subgroups of patients who show early improvement are different at baseline than those who do not, and in what way this relates to ultimate treatment outcome.
Model Results
1. Baseline Model
o Accuracy: 0.6436
o AUC: 0.5304
o F1-Score: 0.2439
o NPV: 0.7320
o PPV: 0.2884
o Sensitivity: 0.2112
o Specificity: 0.805
2. Trajectory Model
o Accuracy: 0.209
o AUC: 0.3338
o F1-Score: 0.346
o NPV: 0.0
o PPV: 0.2195
o Sensitivity: 0.8181
o Specificity: 0.0
Full model files available on Github
Mehltretter, J. 2023. Vivli_Desvenlafaxine_Models. https://github.com/Aifred-Health/Vivli_Desvenlafaxine_Models.git