9 Multilevel Meta-analysis (Robust Variance Estimation)
Should I use Robust Variance Estimation?
Robust Variance Estimation (RVE) is an additonal step you can take to adjust for dependency in effect sizes. Here’s the reccomended usage:
“MLM and RVE result in different types of variance estimates: RVE estimates the between-study variance, MLM splits this variance in two parts: variance between studies and variance between outcomes within studies.
When 25 or less studies are included, MLM results in slightly underestimated variances. MLM is recommended if there are 50 studies included and if the researcher is interested in the variance estimate at both the case and the study level. If only a limited number of studies is included (i.e. 25 studies) and/or if the research interest does not lie in the study and case-level variance estimate, RVE is the preferred method” ( Moeyaert, Ugille, Beretvas, Ferron, Bunuan, and Noortgatep. 570).
https://doi.org/10.1080/13645579.2016.1252189
install.packages("clubSandwich")
library(clubSandwich)Install the clubSandwich package to apply RVE
## Registered S3 method overwritten by 'clubSandwich':
## method from
## bread.mlm sandwichLet’s just recall what the original result (way back when) that we got was
res <- rma(yi, vi, data=dat)
res##
## Random-Effects Model (k = 51; tau^2 estimator: REML)
##
## tau^2 (estimated amount of total heterogeneity): 0.0179 (SE = 0.0050)
## tau (square root of estimated tau^2 value): 0.1338
## I^2 (total heterogeneity / total variability): 78.33%
## H^2 (total variability / sampling variability): 4.61
##
## Test for Heterogeneity:
## Q(df = 50) = 188.5421, p-val < .0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## 0.0978 0.0224 4.3635 <.0001 0.0539 0.1417 ***
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Now let’s re-run the analysis as a multilevel meta-analysis
res_rve <- rma.mv(yi,vi, random = ~ 1 | Study_ID, data = dat)
res_rve##
## Multivariate Meta-Analysis Model (k = 51; method: REML)
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2 0.0193 0.1388 17 no Study_ID
##
## Test for Heterogeneity:
## Q(df = 50) = 188.5421, p-val < .0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## 0.1252 0.0372 3.3651 0.0008 0.0523 0.1982 ***
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1We can apply RVE using the coef_test function
coef_test(res_rve, vcov = "CR2")## Coef. Estimate SE t-stat d.f. (Satt) p-val (Satt) Sig.
## intrcpt 0.125 0.0371 3.37 15.7 0.00397 **While it didn’t make much of a differences let’s try it with a moderator.
Multilvel model
res_rve <- rma.mv(yi, vi, mods = ~ relevel(factor(Branch), ref="Management"), random =~ 1 | Study_ID, data=dat)
res_rve##
## Multivariate Meta-Analysis Model (k = 51; method: REML)
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2 0.0117 0.1080 17 no Study_ID
##
## Test for Residual Heterogeneity:
## QE(df = 46) = 125.8465, p-val < .0001
##
## Test of Moderators (coefficients 2:5):
## QM(df = 4) = 45.5350, p-val < .0001
##
## Model Results:
##
## estimate se
## intrcpt 0.2313 0.0366
## relevel(factor(Branch), ref = "Management")Facilitation -0.1365 0.0314
## relevel(factor(Branch), ref = "Management")Perception -0.1530 0.0328
## relevel(factor(Branch), ref = "Management")Total -0.1695 0.0449
## relevel(factor(Branch), ref = "Management")Understanding -0.1797 0.0308
## zval pval
## intrcpt 6.3193 <.0001
## relevel(factor(Branch), ref = "Management")Facilitation -4.3539 <.0001
## relevel(factor(Branch), ref = "Management")Perception -4.6660 <.0001
## relevel(factor(Branch), ref = "Management")Total -3.7726 0.0002
## relevel(factor(Branch), ref = "Management")Understanding -5.8385 <.0001
## ci.lb ci.ub
## intrcpt 0.1596 0.3030 ***
## relevel(factor(Branch), ref = "Management")Facilitation -0.1980 -0.0751 ***
## relevel(factor(Branch), ref = "Management")Perception -0.2172 -0.0887 ***
## relevel(factor(Branch), ref = "Management")Total -0.2575 -0.0814 ***
## relevel(factor(Branch), ref = "Management")Understanding -0.2401 -0.1194 ***
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Apply RVE
coef_test(res_rve, vcov = "CR2")## Coef. Estimate SE
## intrcpt 0.231 0.0425
## relevel(factor(Branch), ref = "Management")Facilitation -0.137 0.0758
## relevel(factor(Branch), ref = "Management")Perception -0.153 0.0477
## relevel(factor(Branch), ref = "Management")Total -0.169 0.0460
## relevel(factor(Branch), ref = "Management")Understanding -0.180 0.0640
## t-stat d.f. (Satt) p-val (Satt) Sig.
## 5.44 9.48 < 0.001 ***
## -1.80 2.40 0.19167
## -3.21 2.05 0.08195 .
## -3.68 6.40 0.00917 **
## -2.81 2.20 0.09598 .So we can see that RVE can signficantly change our findings