statsmodels.stats.diagnostic.compare_vuong

statsmodels.stats.diagnostic.compare_vuong(results1, results2, correction='none', alternative='two-sided')

Vuong test for non-nested likelihood models.

Added in version 0.15.1.

Parameters:

results1Results

Results instance of the first model. The model needs to provide the log-likelihood of each observation (loglikeobs), for example Logit, Probit, Poisson, NegativeBinomial or ZeroInflatedPoisson.

results2Results

Results instance of the second model, estimated on the same observations and with the same dependent variable as the first model.

correction{“none”, “aic”, “bic”}, optional

Correction for the number of estimated parameters of the two models. “aic” is the Akaike and “bic” is the Schwarz type correction of the log-likelihood ratio. The default is “none”, which applies no correction.

alternative{“two-sided”, “larger”, “smaller”}, optional

Alternative hypothesis. “larger” tests whether the first model is closer to the true distribution than the second model, “smaller” tests the opposite direction. The default is “two-sided”.

Returns:

VuongTestResult

A NamedTuple with fields:

statisticfloat

Vuong z-statistic. Positive values favor the first model.

pvaluefloat

P-value for the chosen alternative.

Notes

The null hypothesis is that both models are equally close to the true data generating process, measured by the Kullback-Leibler divergence. With \(m_i = \log f_1(y_i | x_i) - \log f_2(y_i | x_i)\) the difference of the log-likelihood contributions of observation \(i\), the statistic is

\[V = \frac{\sqrt{n}\,\bar{m}}{\hat{\omega}}, \qquad \hat{\omega}^2 = \frac{1}{n - 1}\sum_{i=1}^n (m_i - \bar{m})^2 ,\]

which is asymptotically standard normal under the null hypothesis [1]. Vuong’s original definition of \(\hat{\omega}^2\) divides by \(n\) instead of \(n - 1\). The difference is asymptotically negligible and the version here is the one used by the vuong function of the R package pscl. The vuongtest function of the R package nonnest2 divides by \(n\), so that its statistic is larger than the one here by the factor \(\sqrt{n / (n - 1)}\).

The corrected versions are the Schwarz and Akaike type corrections of Section 6 of [1], see also [2] for their use in practice. They replace the log-likelihood ratio \(\sum_i m_i\) in the numerator by \(\sum_i m_i - (k_1 - k_2)\) for “aic” and \(\sum_i m_i - (k_1 - k_2) \log(n) / 2\) for “bic”, where \(k_1\) and \(k_2\) are the number of estimated parameters len(results.params). This counts every estimated parameter, including the dispersion parameter of a negative binomial model. The vuong function of pscl does not count the dispersion parameter of a negative binomial model, so that the corrected statistics differ from the pscl values if a negative binomial model is involved.

The uncorrected statistic and the corrected statistics for models with the same parameter counts, for example Poisson against zero-inflated Poisson or Logit against Probit, agree with the results of vuong in pscl. pscl reports the one-sided p-value in the direction of the preferred model.

The test is only valid for non-nested (or overlapping but not nested) models, and it assumes that both models are not equivalent, so that \(\omega^2 > 0\). Statsmodels does not check whether the models are nested. Vuong’s preliminary test of the null hypothesis \(\omega^2 = 0\), which is needed for nested or overlapping models, is not performed. Using the test for nested models, for example Poisson against negative binomial, or for zero-inflation, should be done with care, see [3].

If the log-likelihood contributions of the two models are exactly equal, so that the estimated \(\hat{\omega}\) is exactly zero, the statistic is not defined and both the statistic and the p-value of the result are NaN. No warning is issued. Models that are only numerically almost identical, for example the same model estimated with different optimizers, produce a finite statistic that is meaningless.

References

[1] (1,2)

Vuong, Quang H. 1989. “Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses.” Econometrica 57 (2): 307-333.

[2]

Desmarais, Bruce A., and Jeffrey J. Harden. 2013. “Testing for Zero Inflation in Count Models: Bias Correction for the Vuong Test.” The Stata Journal 13 (4): 810-835.

[3]

Wilson, Paul. 2015. “The Misuse of the Vuong Test for Non-Nested Models to Test for Zero-Inflation.” Economics Letters 127: 51-53.

Examples

Compare a Logit and a Probit model for the same binary outcome.

>>> import statsmodels.api as sm
>>> from statsmodels.stats.api import compare_vuong
>>> data = sm.datasets.spector.load()
>>> exog = sm.add_constant(data.exog)
>>> res_logit = sm.Logit(data.endog, exog).fit(disp=False)
>>> res_probit = sm.Probit(data.endog, exog).fit(disp=False)
>>> result = compare_vuong(res_logit, res_probit, correction="aic")
>>> statistic, pvalue = result
>>> print(f"statistic={statistic:.4f}, pvalue={pvalue:.4f}")
statistic=-0.4972, pvalue=0.6190

The statistic is small and the p-value is large, so the test does not distinguish between the two models.