Lead Investigator: Frank Doyle, Royal College of Surgeons Ireland
Title of Proposal Research: Do psychometrics matter? The effects of advanced psychometric analyses on depression randomised trial outcomes
Vivli Data Request: 8244
Funding Source: None
Potential Conflicts of Interest: None
Summary of the Proposed Research:
Depression is currently the most common mental health disorder worldwide, with an estimated 5% of the population – or 280 million people – suffering this disorder. Depression can affect how you feel, think and behave and can lead to a range of emotional and physical problems. Depression can be a significant health condition, which can inform poor interpersonal relationships, poor work functionality and, at its worst, can inform suicide ideation.
Symptoms caused by depression can vary from person to person and there are a number of effective treatments to help combat depression. These treatments can be broadly categorised as psychotherapies (talking therapy) that can help a person manage their problems by changing the way they think and behave, and pharmacotherapies, such as taking antidepressant medication. The effectiveness of these treatments is typically examined using psychometric assessment, essentially questionnaires which evaluate patients’ experiences of a range of depressive symptoms. However, despite several advanced and very complex statistical methods being developed to evaluate these questionnaires, it is unclear if using these complex techniques are important for finding changes that matter to patients and clinicians.
A recent analysis of 101 studies of 11,910 people with mostly moderate-severe depression, had some very important findings. Firstly, combination psychotherapy (talking therapy) and antidepressants were more effective than either alone. Secondly, antidepressants and psychotherapies did not differ in effects from each other. These types of multi-study analyses allow for estimation of treatment effects that cannot be achieved by single studies alone. In essence, they show that something really ‘matters’ at a population level in terms of treatment effects.
In this current study we propose to obtain already-collected data from thousands of patients who have already participated in depression treatment studies and combine into a large database. This database will be analysed using advanced psychometric techniques, and the results of these complex techniques compared. Using this large database we aim to determining whether psychometric analysis informs any differences in the original results of these depression treatment studies.
This study could make an important contribution to medical science and patient care by highlighting important differences (including gender differences) in depression symptoms and treatments, and potentially improving the accuracy with which depression is psychometrically assessed.
Statistical Analysis Plan:
Data Management
All studies included in analyses will be assigned a unique identifier (e.g. “001”, “002”, “003” etc.) in order to maintain the structure and independence of relative study data. Two dataframes will be established; one for HRSD and one for MADRS data. Each dataframe will only contain data pertaining to respective measures (e.g. the ‘HRSD’ dataframe will only contain HRSD scores at baseline and outcome) and indicated independent variables (trial arm, age, study country and gender). All additional measures and independent variables will be omitted for parsimony. This study will be conducted using complete case analyses, so listwise deletion will be used to remove missing values from each dataframe.
Establish psychometric models of depression pre- (baseline) and post-treatment (outcome), using the HRSD and MADRS
Factor models will be ascertained for combined pre-treatment (baseline) and separately for intervention and comparator outcome groups. For each analysis (i.e., for the HRSD and MADRS), data will be split randomly into two groups to explore dimensionality. Parallel analysis will be applied to both the HRSD and MADRS to determine the dimensionality of respective datasets, with Mokken scaling approaches considered if results are ambiguous. Exploratory factor analysis (EFA) will be conducted to determine the factor structure in each sample, with the resulting model then being subjected to confirmatory factor analysis (CFA) and multidimensional item response theory (MIRT) analytic techniques in the combined total sample for each dataset. The optimum model of depression using the HRSD and MADRS will be identified for each psychometric approach using a number of model selection criteria and fit indices, according to established practices. Model selection criteria will include Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC). Absolute fit indices will include Root Mean Square Error of Approximation (RMSEA) and Standardised Root Mean Square Residual (SRMR), which are considered acceptable at < 0.08. For CFA, relative fit indices will include Comparative Fit Index (CFI) and Tucker-Lewis Index (TLI), which are considered acceptable at > 0.95. Multivariate normality of the data will be examined using the Henze-Zirkler test (Rosseel 2022). This will determine whether a robust maximum likelihood method will be required when estimating model parameters.
Potential gender effects will be explored using measurement invariance methods. Weak invariance will be examined by constraining factor loadings across gender groups, while strong invariance will be examined by constraining factor loadings and intercepts. The presence or absence of gender invariance will then be established by conducting a chi-squared difference test comparing the initial and constrained models. Statistically significant differences will be explored for differences that would be considered clinically significant. In this regard, differential item functioning will be explored by examining a range of statistics, such as factor loading scores, r-squared values, mean values, standard deviations and standard errors. If gender variance is found, individual psychometrically optimised models will be explored using data for men and women separately.
The models produced at baseline and outcome will be examined to identify any potential differences in model structure or item functioning. This will include examination of differences in nonperforming (i.e., dropped) items, examination of the factors or dimensions onto which each item loads and inspection of the salience of each item according to their respective factor loading scores or ability scores. Factor score weights will be garnered from the model that performs best and used to obtain factor scores for each trial for use in later analyses. For the CFA model, factor scores will be calculated by using the “lavPredict” function in Lavaan to estimate unstandardised factor scores for each observed variable. These will then be summed to provide total factor scores for each participant. For the MIRT model, factor scores will be estimated as expected a-posteriori sum (EAPsum) scores using the “fscores” function in the MIRT package. As EAPsum scores are only provided in a standardised format, these will be unstandardised for further analyses using the formula X=Mean+(Z*SD), where X is the unstandardised sum score, Z is the standardised sum score and SD is the standard deviation.
Modelling change and calculating effect sizes
Comparing changes in depression factors to changes in original scores for both HRSD and MADRS: Continuous Outcomes (Primary Outcomes)
A sequence of analyses will be performed to establish whether there is any evidence that the application of factor scores modifies the obtained effect sizes. First, we will establish the overall effects of intervention versus control in the pooled data, using original (raw score) HRSD and MADRS scores, by using a multilevel linear regression model, predicting outcome HRSD and MADRS scores, with treatment group as the predictor, adjusting for baseline scores, with study as the random intercept (Model 1). Then, a similar model will be built, except the total factor score at outcome (i.e. psychometrically-informed outcome) will be modelled, adjusting for total factor score at baseline (Model 2). We will also re-run these two models (i.e., Models 1 & 2), additionally adjusting for participant age and sex (Model 3 & 4 respectively). To facilitate appraisal, effects will also be transformed into Cohen’s d standardised mean difference (SMD) to allow for effect size comparison with previous work.
Furthermore, each trial will be explored individually, to assess changes per trial, and results combined in a meta-analysis. Linear regression analysis will be used separately for the original trial data and factor score data to assess potential changes in outcome scores adjusting for baseline scores. Effect sizes will be obtained for each trial, comparing the original effect sizes from the HRSD and MADRS total scores versus respective factor-informed scores. The difference between the (raw score) SMD and the factor-informed SMD is considered the effect of interest for each trial.
Differences in effect sizes from the original and factor score results will be calculated and explored using random-effects meta-analysis to determine an overall change in effect size for the psychometrically-optimised data. We will also explore potential moderators of the found effects, including study country and gender, in a meta-regression analysis.
Comparing proportions achieving remission (Secondary outcome)
Clinically important differences between original and psychometrically-informed data will also be explored by examining for potential changes in the number and percentage of participants achieving remission, as well as differences in absolute risk reduction. For the HRSD, these analyses may be conducted using a remission threshold of ≤ 7 – which is 14% of the maximum possible score – and then repeated using the psychometrically-determined threshold mentioned above (i.e., 14% of the maximum possible factor score). Remission rates and absolute risk differences will then be examined to identify potential differences in the number of participants achieving remission according to the original and psychometrically-informed data. For the MADRS, a remission threshold of ≤ 4 – which is 15% of the maximum possible score – will be used, with a psychometrically-determined threshold calculated as per the HRSD.
Similar to the primary analysis, overall multilevel models will be built, this time using logistic regression analysing the participants achieving remission, using the raw data and the psychometrically-informed data. Further models will also adjust for age and sex. Then, we will use similar procedures to obtain the numbers achieving remission per trial, which will ultimately be modelled using random effects meta-analysis, with meta-regression exploring the potential moderators.
Requested Studies:
A Multi-centre, Randomised, Double-blind, Parallel Active-controlled Study Evaluating the Efficacy, Safety and Tolerability of Bupropion Hydrochloride Extended-release (Bupropion XL 300mg Once Daily), Escitalopram Oxalate (Escitalopram, 10mg-20mg Once Daily) in Subjects With Major Depressive Disorder
Data Contributor: GlaxoSmithKline
Study ID: NCT02191397
Sponsor ID: 114589
A Phase 3, Randomized, Placebo-Controlled, Double-Blind, Parallel-Design Study to Evaluate the Short-Term, Fixed Dose Efficacy and Safety of LY110140 Once Daily Dosing in Japanese Patients With Major Depressive Disorder
Data Contributor: Lilly
Study ID: NCT01808612
Sponsor ID: 14595
A Phase IV, Multicenter, Randomized, 8-Week, Double-Blind, Placebo-Controlled, Parallel-Group Study to Evaluate the Efficacy of 2 Fixed Doses (50 and 100 mg/Day) of Desvenlafaxine Succinate Sustained-Release (DVS SR) in Adult Outpatients With Major Depressive Disorder
Data Contributor: Pfizer
Study ID: NCT01432457
Sponsor ID: B2061028
A Six Week Randomized, Double-blind, Multi-center, Placebo-controlled, Exploratory, Adaptive Design Study to Explore the Antidepressant Properties of the p38 MAP Kinase Inhibitor GW856553 Compared to Placebo in Adult Subjects With Major Depressive Disorder
Data Contributor: GlaxoSmithKline
Study ID: NCT00976560
Sponsor ID: PKI113009
A Randomised, Double-Blind, Parallel-Group, Placebo-Controlled Study Evaluating the Efficacy and Safety of GSK163090 in Subjects With Major Depressive Disorder
Data Contributor: GlaxoSmithKline
Study ID: NCT00896363
Sponsor ID: 109035
A Multicenter, Double-Blind, Placebo-Controlled, Randomized Withdrawal, Parallel Group Study To Evaluate The Efficacy And Safety Of 50 mg/Day Of DVS SR In Adult Outpatients With Major Depressive Disorder
Data Contributor: Pfizer
Study ID: NCT00887224
Sponsor ID: 3151A1-3360
A Randomized, Double-blind, Placebo Controlled Trial to Evaluate the Clinical Effects of Controlled Release Paroxetine in the Treatment of Major Depressive Disorder
Data Contributor: GlaxoSmithKline
Study ID: NCT00866294
Sponsor ID: 112810
A Multicenter, Randomized, Double-Blind, Placebo-Controlled, Parallel-Group Study To Evaluate The Efficacy And Safety Of 2 Fixed Doses (10 And 50 mg/Day) Of DVS SR Tablets In Adult Outpatients With Major Depressive Disorder
Data Contributor: Pfizer
Study ID: NCT00863798
Sponsor ID: 3151A1-3362
A Multicenter, Double-Blind, Randomized, Placebo-Controlled Study To Evaluate Functional Outcome In Outpatients With Major Depressive Disorder Treated With Desvenlafaxine Succinare Sustained Release
Data Contributor: Pfizer
Study ID: NCT00824291
Sponsor ID: 3151A1-4415
Comparison of Two Different Treatment Strategies in Patients With Major Depressive Disorder Not Exhibiting Improvement on Escitalopram Treatment: Early vs. Delayed Intervention Strategy
Data Contributor: Lilly
Study ID: NCT00810069
Sponsor ID: 12329
A Multicenter, Randomized, Double-Blind, Placebo-Controlled, Parallel-Group Study to Evaluate the Efficacy and Safety of 2 Fixed Doses (25 and 50 mg/Day) of DVS SR Tablets in Adult Outpatients With Major Depressive Disorder
Data Contributor: Pfizer
Study ID: NCT00798707
Sponsor ID: 3151A1-3359
A Randomized, Double-blind, Parallel-Group, Placebo-Controlled, Fixed-Dose Study Comparing the Efficacy and Safety of 3 Doses of Lu AA21004 in Acute Treatment of Adults With Major Depressive Disorder
Data Contributor: Takeda
Study ID: NCT00735709
Sponsor ID: LuAA21004_305
A Randomized, Double-blind, Parallel-group, Placebo-controlled, Fixed-dose Study Comparing the Efficacy and Safety of Lu AA21004 Versus Placebo in Acute Treatment of Adults With Major Depressive Disorder
Data Contributor: Takeda
Study ID: NCT00672958
Sponsor ID: LuAA21004_303
A Randomized, Double Blind, Placebo Controlled Study to Explore the Antidepressant Properties of P38a Kinase Inhibitor GW856553X 15mg Compared to PBO in Subjects With Major Depressive Disorder Exhibiting Symptoms of Loss of Energy and Interest and Psychomotor Retardation, for a Six Week Treatment Period
Data Contributor: GlaxoSmithKline
Study ID: NCT00569062
Sponsor ID: PKI108574
Duloxetine Versus Placebo in Patients With Major Depressive Disorder (MDD): Assessment of Energy and Vitality in MDD
Data Contributor: Lilly
Study ID: NCT00536471
Sponsor ID: 11669
A Multicenter, Randomized, 8-week, Double-blind, Placebo-controlled Study Followed by a 6-month Open-label Extension to Evaluate the Efficacy and Safety of DVS SR in Peri- and Postmenopausal Women With Major Depressive Disorder
Data Contributor: Pfizer
Study ID: NCT00369343
Sponsor ID: 3151A1-403
A Multicenter, Randomized, Double-Blind, Placebo-Controlled, Parallel-Group Study to Evaluate the Efficacy and Safety of 2 Fixed Doses (50 mg, 100 mg) of Desvenlafaxine Sustained-Release Tablets in Adult Outpatients With Major Depressive Disorder
Data Contributor: Pfizer
Study ID: NCT00300378
Sponsor ID: 3151A1-333
A Multicenter, Randomized, Double-Blind, Placebo-Controlled, Parallel Group Study To Evaluate The Efficacy And Safety Of Two Fixed Doses (200 Mg, Or 400 Mg) Of DVS-233 SR In Adult Outpatients With Major Depressive Disorder
Data Contributor: Pfizer
Study ID: NCT00073762
Sponsor ID: 3151A1-308?
A Multicenter, Randomized, Double-Blind, Placebo-Controlled, Parallel Group Study To Evaluate The Efficacy And Safety Of Three Fixed Doses (100 Mg, 200 Mg, Or 400 Mg) Of DVS-233 SR In Adult Outpatients With Major Depressive Disorder
Data Contributor: Pfizer
Study ID: NCT00072774
Sponsor ID: 3151A1-306
A Randomized, Double-Blind, Parallel-Group, Placebo-Controlled Fixed Dose Study Evaluating the Efficacy and Safety of Paroxetine CR in Elderly Outpatients Diagnosed With Major Depressive Disorder
Data Contributor: GlaxoSmithKline
Study ID: NCT00067444
Sponsor ID: 29060/874
A Multicenter, Randomized, Double-blind, Placebo-Controlled, Parallel-Group, Efficacy and Safety Study of a Flexible Dose of DVS-233 SR in Adult Outpatients With Major Depressive Disorder
Data Contributor: Pfizer
Study ID: NCT00063206
Sponsor ID: 3151A1-304
Duloxetine Versus Placebo in the Treatment of Elderly Patients With Major Depressive Disorder
Data Contributor: Lilly
Study ID: NCT00062673
Sponsor ID: 6091
An 8-Week, Randomized, Double-Blind, Placebo-Controlled, Multicenter, Fixed-Dose Study Comparing the Efficacy and Safety of GW597599B or Paroxetine to Placebo in Moderately to Severely Depressed Patients With Major Depressive Disorder
Data Contributor: GlaxoSmithKline
Study ID: NCT00048204
Sponsor ID: NKD20006
Duloxetine Versus Duloxetine Plus Non-Pharmacological Intervention in the Treatment of Depression
Data Contributor: Lilly
Study ID: NCT00494377
Sponsor ID: 8601
Duloxetine Versus Placebo in the Treatment of Fibromyalgia Patients With or Without Major Depressive Disorder
Data Contributor: Lilly
Study ID: NCT00489073
Sponsor ID: 6158
Duloxetine Versus Placebo in the Long-Term Treatment of Patients With Late-Life Major Depression
Data Contributor: Lilly
Study ID: NCT00406848
Sponsor ID: 10815
Protocol F1J-US-HMBY Dose Escalation, Double-Blind Treatment With Duloxetine Hydrochloride Once Daily Dosing for Evaluation of Safety in Major Depression
Data Contributor: Lilly
Study ID: NCT00042575
Sponsor ID: 6475
Duloxetine Hydrochloride (LY248686) Protocol F1J-US-HMCB Duloxetine Once-Daily Dosing Versus Placebo in Patients With Major Depression and Pain
Data Contributor: Lilly
Study ID: NCT00036335
Sponsor ID: 6353
A Randomized, Double-Blind, Placebo-Controlled, Parallel-Group, Phase III Study to Evaluate the Efficacy and Safety of Once Daily Oral Lu AA21004 in Patients With Major Depressive Disorder
Data Contributor: Takeda
Study ID: NCT02389816
Sponsor ID: LuAA21004/CCT-004
A Randomized, Double-Blind, Placebo-Controlled, Phase 4, Relapse Prevention Study Evaluating the Efficacy and Safety of Vortioxetine (5, 10 and 20 mg) in Adults With Major Depressive Disorder
Data Contributor: Takeda
Study ID: NCT02371980
Sponsor ID: LuAA21004_402
A Randomized, Double-Blind, Placebo-Controlled, Phase 3 Study to Evaluate the Efficacy and Safety of Once a Day, TAK-375 (Ramelteon) Tablet for Sublingual Administration (TAK-375SL Tablet) as an Adjunctive Therapy in the Treatment of Acute Depressive Episodes Associated With Bipolar 1 Disorder in Adult Subjects
Data Contributor: Takeda
Data Request ID: 00008244
Sponsor ID: TAK-375SL_301
A Randomized, Double-Blind, Placebo-Controlled, Parallel-Group, Phase III Study to Assess the Efficacy and Safety of Lu AA21004 in Patients With Major Depressive Disorder
Data Contributor: Takeda
Study ID: NCT01355081
Sponsor ID: LuAA21004/CCT-003
A Multinational, Randomized, Double-Blind, Placebo-Controlled, Dose Ranging Study to Assess the Efficacy and Safety of Lu AA21004 in Patients With Major Depressive Disorder
Data Contributor: Takeda
Study ID: NCT01255787
Sponsor ID: LuAA21004/CCT-002
A Phase 3, Randomized, Double-Blind, Parallel-Group, Placebo-Controlled, Fixed-Dose Study Comparing the Efficacy and Safety of 2 Doses (10 and 15 mg) of Lu AA21004 in Acute Treatment of Adults With Major Depressive Disorder
Data Contributor: Takeda
Data Request ID: 00008244
Sponsor ID: LuAA21004_317
A Phase 3, Randomized, Double-Blind, Parallel-Group, Placebo-Controlled, Fixed-Dose Study Comparing the Efficacy and Safety of 2 Doses (10 and 20 mg) of Lu AA21004 in Acute Treatment of Adults With Major Depressive Disorder
Data Contributor: Takeda
Study ID: NCT01163266
Sponsor ID: LuAA21004_316
A Phase 4, 8-week, Double-blind, Randomized Study Comparing Switching to Duloxetine or Escitalopram in Patients With Major Depressive Disorder and Residual Apathy in the Absence of Depressed Mood
Data Contributor: Lilly
Study ID: NCT00985504
Sponsor ID: 13018
A Multicenter, Randomized, Double-Blind, Placebo-Controlled, Duloxetine-Referenced, Parallel-Group Study to Evaluate the Efficacy and Safety of 2 Fixed Doses (50mg, 100mg) of Desvenlafaxine Sustained-Release Tablets in Adult Outpatients With Major Depressive Disorder
Data Contributor: Pfizer
Study ID: NCT00384033
Sponsor ID: 3151A1-335
Study AK1113351, a Fixed Dose Study of 323U66 SR in the Treatment of Major Depressive Disorder (MDD) – a Multi-center, Placebo-controlled, Randomized, Double-blind, Parallel-comparison Study
Data Contributor: GlaxoSmithKline
Study ID: NCT01138007
Sponsor ID: 113351
Duloxetine Versus Placebo in the Acute Treatment of Patients With Major Depressive Disorder and Associated Painful Physical Symptoms
Data Contributor: Lilly
Study ID: NCT01070329
Sponsor ID: 13630
A Phase 4, 8-Week, Double-Blind, Randomized, Placebo-Controlled Study Evaluating the Efficacy of Duloxetine 60 mg Once Daily in Outpatients With Major Depressive Disorder and Associated Painful Physical Symptoms
Data Contributor: Lilly
Study ID: NCT01000805
Sponsor ID: 13399
The Study of Olanzapine Plus Fluoxetine in Combination for Treatment-Resistant Depression Without Psychotic Features
Data Contributor: Lilly
Study ID: NCT00035321
Sponsor ID: 6272
A Multicenter, Parallel-Group, Randomized, 10-Week, Double-Blind, Placebo-Controlled Study To Evaluate The Efficacy And Safety Of 50 mg Of DVS SR In The Treatment Of Peri- And Postmenopausal Women With Major Depressive Disorder
Data Contributor: Pfizer
Study ID: NCT01121484
Sponsor ID: 3151A1-3364
Public Disclosures:
- Doyle, F. Byrne, D., Boland F. 2023. Do psychometrics matter? The effects of advanced psychometrics on antidepressant randomised trial outcomes. International Society for Affective Disorders. Abstract O-02. pg.47
- Doyle, F. Byrne, D., Boland F. 2024. Do psychometrics matter? The effects of psychometrics on depression trial outcomes. International Meeting of the Psychomtric Society. Abstract. pg.377