Susceptibility of Synthetic Control Methods to Reference Category Choice

Abstract

While generally treated as arbitrary, the choice of which category of a categorical or compositional covariate to omit as a reference in a synthetic control analysis can substantially influence the resulting causal estimate. Synthetic control methods estimate causal effects by constructing synthetic counterfactual units to compare to treated units. These synthetic counterfactuals are constructed by taking a weighted combination of non-treated units such that the treated and synthetic units match as closely as possible on observed characteristics in the pre-treatment period. When the observable characteristics of interest include categorical or compositional variables, researchers tend to omit a reference category, in part because the process of identifying a synthetic control typically relies on a regression of the outcome on observed covariates. While the choice of reference category is generally treated as arbitrary, we show that this choice affects the relative weight of each covariate in determining the synthetic control, in turn affecting the ultimate causal estimate. In order to assess the severity of the resulting uncertainty, we conduct a calibrated simulation study drawing from Current Population Survey data. In addition to a continuous outcome and one continuous covariate, we simulate three compositional covariates – race, education, and industry – with three, four, and five categories respectively. We then generate synthetic control estimates for the 60 possible reference category combinations with various specifications of the Synth, Augsynth, Gsynth, and CausalImpact R packages. The resulting causal effect distributions provide an assessment of uncertainty due to reference category choice across various implementation strategies. Our preliminary results suggest that reference category uncertainty is not negligible, varying by as much as 10% of the average estimate, or 75% of the standard deviation of the outcome.

Date
Location
Berkeley, CA