Testing alternative models

However, even if the the model is incorrect, it may be that a correct model would still give show a correlation between the carry laws and less crime. A study by Bartley and Cohen [4] sheds some light on this possibility. They reran the regressions with thousands of different models (formed by deleting variables from Lott's model and by adding a trend term). If the carry law is associated with a reduction in crime in all of these models then we might conclude that is associated with a reduction in the correct model, without us having to identify which of the models is the correct one.

Figures 1-4 of their paper show that the carry laws were not consistently associated with a crime reduction in any crime category: that is, there were some models where the law was associated with an increase for each crime category studied. I should note, however, that if we restrict things to just models that include a trend component, homicide and robbery show consistent reductions. For this reason, Bartley and Cohen argued that Lott's results should not be dismissed as unfounded.

Dezhbakhsh and Rubin [10,11] re-examined the data using a more general model that allowed the carry law to have different effects in each county and to affect other parameters in the model. With this model they found the carry law did not have any clear effect on rape or assault, that it was associated with a reduction in homicide in six out of 33 states, and with an increase in robbery in 13 out of out of 33 states. The evidence here is stronger for an increase than for a decrease.

Plassmann and Tideman [39] point out that Lott's analysis technique assumes that crime rates are normally distributed and that this is not even close to being true for low crime counties. When they made some plausible changes to the specification, the effects on murder vanished. However, when they did their own analysis assuming that the murder rate was Poisson distributed, they found an even stronger effect (a 12% decrease). They also looked at the effects on each state and found a confusing pattern of results, with the effect varying from a statistically significant increase of 6.5% (Virginia) to a statistically significant decrease of 35% (Montana). While we would not expect the laws to have exactly the same effect in every state, it seems hard to see how the effects could be so radically different.

Duggan [12] points out another problem with Lott's analysis:

One problem with these regression estimates is that Lott and Mustard are implicitly assuming that these laws are varying at the county level, when in fact they are varying only at the state level.
The reason this is a problem is that you would expect crime rates in counties within the same state to be correlated. This problem does not bias the estimates of the law's effect, but causes the standard errors to be underestimated, so that some results may appear to be statistically significant when they are not.

On page 278, note 3, Lott comments on this problem, but erroneously claims that including dummy variables for all counties solves the problem. This is clearly false. The dummy variables only account for fixed differences between counties and do not address the within-state correlations between counties.

After adjustments to account for this problem, Duggan found that

none of the coefficient estimates on the CCW variable remain statistically significant.

Lott's response to Duggan's paper was to repeat his false claim:

The correlation of the error terms across counties is picked up when one has county fixed effects included in the regression. He does not do the adjustment recognizing that the county fixed effects are already picking up what he wants to adjust for. [25]

Moody [35] noticed the same problem as Duggan:

Merging an aggregate variable with microlevel variables causes ordinary least squares formulas to severely overestimate the t-ratios associated with the aggregate variables. ...I reestimated the model using the original county-level data set but adjusted the standard errors for clustering within states. The results were somewhat different from the original Lott and Mustard findings. ...While shall-issue laws reduce violent crime in general in all models, the effects seem to be concentrated in robbery. Murder and rape are significantly reduced in only one version of the model.

In Lott's response to Moody [29] he still did not admit to making a mistake but rather stated that he ``had already discussed this issue''.

Tim Lambert