Prepare for a really dorky blog entry.
I was very struck by Kendler, Kuhn, & Prescott's 2004 article--not necessarily because of their conclusions about risk factors for depressive episodes but because of how they got there. In other words, I appreciated their approach to various statistical issues, especially model building and the use of raw probabilities. Admittedly, I am still a quantitative infant, but I particularly appreciated their deliberate decision to compare additive versus multiplicative models of risk. It is often tempting to try to fit any dataset to a linear model, not for any theoretical reason, but simply for reasons of convenience. Much can be gained, however, from considering the implications of fitting data to different mathematical functions; in this case, for instance, fitting a multiplicative model of risk suggests processes that differ in important ways from the ones that would operate in an additive model of risk. Just because linear models are easy to construct and analyze does not mean that they are the best way to understand our data!
I also appreciated Kendler et al.'s discussion of their decision to use raw probabilities as opposed to transforming them, perhaps to log(raw probability). Using a logarithmic transformation may indeed have made the process of statistical analysis easier but it also would have disguised one of the most interesting features of these data, which is the nonlinear relationship between risk of a depressive episode and contextual threat X neuroticism. I agree with the "public health argument" and I also think that it would have been much more difficult to interpret log(hazard ratio) than it is to interpret the raw probabilities. Despite the statistical convenience that transformation to a log scale can provide, it can lead to results that are difficult to interpret. Here, sticking with the raw probabilities makes the patterns Kendler et al. wish to highlight much more clear.
P.S. Jim--you wanted me to remind you to get me the reference(s) on doing meta-analyses of single-case studies.
Subscribe to:
Post Comments (Atom)
1 comment:
I think if we are going to really explain the variance we have left in clinical psychology (which is ample), we are going to have to start thinking in multiplicative terms a lot of the time. Ah, those halcyon days of yore, when simple linear models seemed to really do the trick--gone forever.
On the topic of meta-analysis for case studies, check this out, found at the following website:
http://www2.chass.ncsu.edu/garson/pa765/cases.htm
Meta-Analysis is a particular methodology for extending grounded theory to a number of case studies. In meta-analysis the researcher creates a meta-analytic schedule, which is a cross-case summary table in which the rows are case studies and the columns are variable-related findings or other study attributes (ex., time frame, research entity, case study design type, number and selection method for interviewees, threats to validity like researcher involvement in the research entity). The cell entries may be simple checkmarks indicating a given study supported a given variable relationship, or the cell entries may be brief summaries of findings on a given relationship or brief description of study attributes. The purpose of meta-analysis is to allow the researcher to use the summary of case studies reflected in the meta-analytic table to make theoretical generalizations. In doing so, sometimes the researcher will weight the cases according to the number of research entities studied, since some case studies may examine multiple entities. See Hodson (1999); Jensen and Rodgers (2001: 239 ff.). Hodson (1999: 74-80) reproduces an example of a meta-analytic schedule for the topic of workplace ethnography.
Problems of meta-analysis include what even case study advocates admit is the "formidible challenge" (Jensen and Rodgers, 2001: 241) involved in developing a standardized meta-analytic schedule which fits the myriad dimensions of any sizeable number of case studies. No widely accepted "standardized" schedules exist. Moreover, for any given proposed schedule, many or most specific case studies will simply not report findings in one or more of the column categories, forcing meta-analysts either to accept a great deal of missing data or to have to do additional research by contacting case authors or even case subjects.
Considerations in implementing meta-analytic schedules:
1. Variables: In addition to substantive variables particular to the researcher's subject, methodological variables should be collected, such as date of data collection, subject pool, and methodological techniques employed.
2. Coder training. It is customary to provide formal training for coders, who ideally should not be the researchers so that data collection is separated from data interpretation.
3. Reliability. The researcher must establish inter-rater reliability, which in turn implies there must be multiple raters. Reliability is generally increased through rater debriefing sessions in which raters are encouraged to discuss coding challenges. Duplicate coding (allowing 10% or so of records to be coded by two coders rather than one) is also used to track reliability. In larger projects, rating may be cross-validated across two or more groups of coders.
4. Data weighting. Meta-analysis often involves statistical analysis of results, where cases are studies. The researcher must decide whether cases based on a larger sample size should be weighted more in any statistical analysis. In general, weighting is appropriate when cases are drawn from the same population to which the researcher wishes to generalize.
5. Handling missing data. Dropping cases where some variables have missing data is generally unacceptable unless there are only a very small number of such cases as (1) it is more likely that missing-data cases are related to the variables of the study than that they are randomly distributed, and (2) dropping cases when the number of cases is not large (as is typical of meta-analytic studies) diminishes the power of any statistical analysis. There is no good solution for missing data. See the separate section on data imputation, but maximum likelihood estimation of missing values carries fewer assumptions about data distribution than using regression estimates or substituting means. SPSS supports MLE imputation.
6. Outliers. Metapanalysis often involves results coded from a relatively small number of cases (ex., < 100). Consequently, any statistical analysis may be affected strongly by the presence of outlier cases. Sensitivity analysis should be conducted to understand the difference in statistical conclusions with and without the outlier cases included. The researcher may decide that deviant case analysis may be appropriate, based on a finding that relationships among the variables operate differently for outlier cases.
7. Spatial autocorrelation. It is possible that central tendencies and conclusions emerging from meta-analytic studies will be biased because cases cluster spatially. If many cases are from a spatially neighboring area and if the relationships being studied are spatially related, then generalization to a larger reference area will be biased. If the researcher has included longitude and latitude (or some other spatial indicator) as variables, then many geographic information systems packages and some statistical packages can check for spatial autocorrelation (see Land and Deane, 1992). However, a visual approach of mapping cases to identify clusters, then comparing in-cluster and out-of-cluster statistical results usually is a sufficient check on spatial autocorrelation.
Post a Comment