Weighted Generalized Linear Models

In [1]:
import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
import statsmodels.api as sm

Weighted GLM: Poisson response data

Load data

In this example, we'll use the affair dataset using a handful of exogenous variables to predict the extra-marital affair rate.

Weights will be generated to show that freq_weights are equivalent to repeating records of data. On the other hand, var_weights is equivalent to aggregating data.

In [2]:
print(sm.datasets.fair.NOTE)
::

    Number of observations: 6366
    Number of variables: 9
    Variable name definitions:

        rate_marriage   : How rate marriage, 1 = very poor, 2 = poor, 3 = fair,
                        4 = good, 5 = very good
        age             : Age
        yrs_married     : No. years married. Interval approximations. See
                        original paper for detailed explanation.
        children        : No. children
        religious       : How relgious, 1 = not, 2 = mildly, 3 = fairly,
                        4 = strongly
        educ            : Level of education, 9 = grade school, 12 = high
                        school, 14 = some college, 16 = college graduate,
                        17 = some graduate school, 20 = advanced degree
        occupation      : 1 = student, 2 = farming, agriculture; semi-skilled,
                        or unskilled worker; 3 = white-colloar; 4 = teacher
                        counselor social worker, nurse; artist, writers;
                        technician, skilled worker, 5 = managerial,
                        administrative, business, 6 = professional with
                        advanced degree
        occupation_husb : Husband's occupation. Same as occupation.
        affairs         : measure of time spent in extramarital affairs

    See the original paper for more details.

Load the data into a pandas dataframe.

In [3]:
data = sm.datasets.fair.load_pandas().data

The dependent (endogenous) variable is affairs

In [4]:
data.describe()
Out[4]:
rate_marriage age yrs_married children religious educ occupation occupation_husb affairs
count 6366.000000 6366.000000 6366.000000 6366.000000 6366.000000 6366.000000 6366.000000 6366.000000 6366.000000
mean 4.109645 29.082862 9.009425 1.396874 2.426170 14.209865 3.424128 3.850141 0.705374
std 0.961430 6.847882 7.280120 1.433471 0.878369 2.178003 0.942399 1.346435 2.203374
min 1.000000 17.500000 0.500000 0.000000 1.000000 9.000000 1.000000 1.000000 0.000000
25% 4.000000 22.000000 2.500000 0.000000 2.000000 12.000000 3.000000 3.000000 0.000000
50% 4.000000 27.000000 6.000000 1.000000 2.000000 14.000000 3.000000 4.000000 0.000000
75% 5.000000 32.000000 16.500000 2.000000 3.000000 16.000000 4.000000 5.000000 0.484848
max 5.000000 42.000000 23.000000 5.500000 4.000000 20.000000 6.000000 6.000000 57.599991
In [5]:
data[:3]
Out[5]:
rate_marriage age yrs_married children religious educ occupation occupation_husb affairs
0 3.0 32.0 9.0 3.0 3.0 17.0 2.0 5.0 0.111111
1 3.0 27.0 13.0 3.0 1.0 14.0 3.0 4.0 3.230769
2 4.0 22.0 2.5 0.0 1.0 16.0 3.0 5.0 1.400000

In the following we will work mostly with Poisson. While using decimal affairs works, we convert them to integers to have a count distribution.

In [6]:
data["affairs"] = np.ceil(data["affairs"])
data[:3]
Out[6]:
rate_marriage age yrs_married children religious educ occupation occupation_husb affairs
0 3.0 32.0 9.0 3.0 3.0 17.0 2.0 5.0 1.0
1 3.0 27.0 13.0 3.0 1.0 14.0 3.0 4.0 4.0
2 4.0 22.0 2.5 0.0 1.0 16.0 3.0 5.0 2.0
In [7]:
(data["affairs"] == 0).mean()
Out[7]:
0.6775054979579014
In [8]:
np.bincount(data["affairs"].astype(int))
Out[8]:
array([4313,  934,  488,  180,  130,  172,    7,   21,   67,    2,    0,
          0,   17,    0,    0,    0,    3,   12,    8,    0,    0,    0,
          0,    0,    2,    2,    2,    3,    0,    0,    0,    0,    0,
          0,    0,    0,    0,    0,    0,    1,    1,    0,    0,    0,
          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
          0,    0,    0,    1])

Condensing and Aggregating observations

We have 6366 observations in our original dataset. When we consider only some selected variables, then we have fewer unique observations. In the following we combine observations in two ways, first we combine observations that have values for all variables identical, and secondly we combine observations that have the same explanatory variables.

Dataset with unique observations

We use pandas's groupby to combine identical observations and create a new variable freq that count how many observation have the values in the corresponding row.

In [9]:
data2 = data.copy()
data2['const'] = 1
dc = data2['affairs rate_marriage age yrs_married const'.split()].groupby('affairs rate_marriage age yrs_married'.split()).count()
dc.reset_index(inplace=True)
dc.rename(columns={'const': 'freq'}, inplace=True)
print(dc.shape)
dc.head()
(476, 5)
Out[9]:
affairs rate_marriage age yrs_married freq
0 0.0 1.0 17.5 0.5 1
1 0.0 1.0 22.0 2.5 3
2 0.0 1.0 27.0 2.5 1
3 0.0 1.0 27.0 6.0 5
4 0.0 1.0 27.0 9.0 1

Dataset with unique explanatory variables (exog)

For the next dataset we combine observations that have the same values of the explanatory variables. However, because the response variable can differ among combined observations, we compute the mean and the sum of the response variable for all combined observations.

We use again pandas groupby to combine observations and to create the new variables. We also flatten the MultiIndex into a simple index.

In [10]:
gr = data['affairs rate_marriage age yrs_married'.split()].groupby('rate_marriage age yrs_married'.split())
df_a = gr.agg(['mean', 'sum','count'])
def merge_tuple(tpl):
    if isinstance(tpl, tuple) and len(tpl) > 1:
        return "_".join(map(str, tpl))
    else:
        return tpl
    
df_a.columns = df_a.columns.map(merge_tuple)
df_a.reset_index(inplace=True)
print(df_a.shape)
df_a.head()
(130, 6)
Out[10]:
rate_marriage age yrs_married affairs_mean affairs_sum affairs_count
0 1.0 17.5 0.5 0.000000 0.0 1
1 1.0 22.0 2.5 3.900000 39.0 10
2 1.0 27.0 2.5 3.400000 17.0 5
3 1.0 27.0 6.0 0.900000 9.0 10
4 1.0 27.0 9.0 1.333333 4.0 3

After combining observations with have a dataframe dc with 467 unique observations, and a dataframe df_a with 130 observations with unique values of the explanatory variables.

In [11]:
print('number of rows: \noriginal, with unique observations, with unique exog')
data.shape[0], dc.shape[0], df_a.shape[0]
number of rows: 
original, with unique observations, with unique exog
Out[11]:
(6366, 476, 130)

Analysis

In the following, we compare the GLM-Poisson results of the original data with models of the combined observations where the multiplicity or aggregation is given by weights or exposure.

original data

In [12]:
glm = smf.glm('affairs ~ rate_marriage + age + yrs_married',
              data=data, family=sm.families.Poisson())
res_o = glm.fit()
print(res_o.summary())
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:                affairs   No. Observations:                 6366
Model:                            GLM   Df Residuals:                     6362
Model Family:                 Poisson   Df Model:                            3
Link Function:                    log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -10351.
Date:                Sun, 24 Nov 2019   Deviance:                       15375.
Time:                        07:50:02   Pearson chi2:                 3.23e+04
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
Intercept         2.7155      0.107     25.294      0.000       2.505       2.926
rate_marriage    -0.4952      0.012    -41.702      0.000      -0.518      -0.472
age              -0.0299      0.004     -6.691      0.000      -0.039      -0.021
yrs_married      -0.0108      0.004     -2.507      0.012      -0.019      -0.002
=================================================================================
In [13]:
res_o.pearson_chi2 / res_o.df_resid
Out[13]:
5.078702313363196

condensed data (unique observations with frequencies)

Combining identical observations and using frequency weights to take into account the multiplicity of observations produces exactly the same results. Some results attribute will differ when we want to have information about the observation and not about the aggregate of all identical observations. For example, residuals do not take freq_weights into account.

In [14]:
glm = smf.glm('affairs ~ rate_marriage + age + yrs_married',
              data=dc, family=sm.families.Poisson(), freq_weights=np.asarray(dc['freq']))
res_f = glm.fit()
print(res_f.summary())
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:                affairs   No. Observations:                  476
Model:                            GLM   Df Residuals:                     6362
Model Family:                 Poisson   Df Model:                            3
Link Function:                    log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -10351.
Date:                Sun, 24 Nov 2019   Deviance:                       15375.
Time:                        07:50:02   Pearson chi2:                 3.23e+04
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
Intercept         2.7155      0.107     25.294      0.000       2.505       2.926
rate_marriage    -0.4952      0.012    -41.702      0.000      -0.518      -0.472
age              -0.0299      0.004     -6.691      0.000      -0.039      -0.021
yrs_married      -0.0108      0.004     -2.507      0.012      -0.019      -0.002
=================================================================================
In [15]:
res_f.pearson_chi2 / res_f.df_resid
Out[15]:
5.078702313363196

condensed using var_weights instead of freq_weights

Next, we compare var_weights to freq_weights. It is a common practice to incorporate var_weights when the endogenous variable reflects averages and not identical observations. I don't see a theoretical reason why it produces the same results (in general).

This produces the same results but df_resid differs the freq_weights example because var_weights do not change the number of effective observations.

In [16]:
glm = smf.glm('affairs ~ rate_marriage + age + yrs_married',
              data=dc, family=sm.families.Poisson(), var_weights=np.asarray(dc['freq']))
res_fv = glm.fit()
print(res_fv.summary())
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:                affairs   No. Observations:                  476
Model:                            GLM   Df Residuals:                      472
Model Family:                 Poisson   Df Model:                            3
Link Function:                    log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -10351.
Date:                Sun, 24 Nov 2019   Deviance:                       15375.
Time:                        07:50:02   Pearson chi2:                 3.23e+04
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
Intercept         2.7155      0.107     25.294      0.000       2.505       2.926
rate_marriage    -0.4952      0.012    -41.702      0.000      -0.518      -0.472
age              -0.0299      0.004     -6.691      0.000      -0.039      -0.021
yrs_married      -0.0108      0.004     -2.507      0.012      -0.019      -0.002
=================================================================================

Dispersion computed from the results is incorrect because of wrong df_resid. It is correct if we use the original df_resid.

In [17]:
res_fv.pearson_chi2 / res_fv.df_resid, res_f.pearson_chi2 / res_f.df_resid
Out[17]:
(68.45488160512002, 5.078702313363196)

aggregated or averaged data (unique values of explanatory variables)

For these cases we combine observations that have the same values of the explanatory variables. The corresponding response variable is either a sum or an average.

using exposure

If our dependent variable is the sum of the responses of all combined observations, then under the Poisson assumption the distribution remains the same but we have varying exposure given by the number of individuals that are represented by one aggregated observation.

The parameter estimates and covariance of parameters are the same with the original data, but log-likelihood, deviance and Pearson chi-squared differ

In [18]:
glm = smf.glm('affairs_sum ~ rate_marriage + age + yrs_married',
              data=df_a, family=sm.families.Poisson(), exposure=np.asarray(df_a['affairs_count']))
res_e = glm.fit()
print(res_e.summary())
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:            affairs_sum   No. Observations:                  130
Model:                            GLM   Df Residuals:                      126
Model Family:                 Poisson   Df Model:                            3
Link Function:                    log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -740.75
Date:                Sun, 24 Nov 2019   Deviance:                       967.46
Time:                        07:50:02   Pearson chi2:                     926.
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
Intercept         2.7155      0.107     25.294      0.000       2.505       2.926
rate_marriage    -0.4952      0.012    -41.702      0.000      -0.518      -0.472
age              -0.0299      0.004     -6.691      0.000      -0.039      -0.021
yrs_married      -0.0108      0.004     -2.507      0.012      -0.019      -0.002
=================================================================================
In [19]:
res_e.pearson_chi2 / res_e.df_resid
Out[19]:
7.350789109179526

using var_weights

We can also use the mean of all combined values of the dependent variable. In this case the variance will be related to the inverse of the total exposure reflected by one combined observation.

In [20]:
glm = smf.glm('affairs_mean ~ rate_marriage + age + yrs_married',
              data=df_a, family=sm.families.Poisson(), var_weights=np.asarray(df_a['affairs_count']))
res_a = glm.fit()
print(res_a.summary())
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:           affairs_mean   No. Observations:                  130
Model:                            GLM   Df Residuals:                      126
Model Family:                 Poisson   Df Model:                            3
Link Function:                    log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -5954.2
Date:                Sun, 24 Nov 2019   Deviance:                       967.46
Time:                        07:50:02   Pearson chi2:                     926.
No. Iterations:                     5                                         
Covariance Type:            nonrobust                                         
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
Intercept         2.7155      0.107     25.294      0.000       2.505       2.926
rate_marriage    -0.4952      0.012    -41.702      0.000      -0.518      -0.472
age              -0.0299      0.004     -6.691      0.000      -0.039      -0.021
yrs_married      -0.0108      0.004     -2.507      0.012      -0.019      -0.002
=================================================================================

Comparison

We saw in the summary prints above that params and cov_params with associated Wald inference agree across versions. We summarize this in the following comparing individual results attributes across versions.

Parameter estimates params, standard errors of the parameters bse and pvalues of the parameters for the tests that the parameters are zeros all agree. However, the likelihood and goodness-of-fit statistics, llf, deviance and pearson_chi2 only partially agree. Specifically, the aggregated version do not agree with the results using the original data.

Warning: The behavior of llf, deviance and pearson_chi2 might still change in future versions.

Both the sum and average of the response variable for unique values of the explanatory variables have a proper likelihood interpretation. However, this interpretation is not reflected in these three statistics. Computationally this might be due to missing adjustments when aggregated data is used. However, theoretically we can think in these cases, especially for var_weights of the misspecified case when likelihood analysis is inappropriate and the results should be interpreted as quasi-likelihood estimates. There is an ambiguity in the definition of var_weights because they can be used for averages with correctly specified likelihood as well as for variance adjustments in the quasi-likelihood case. We are currently not trying to match the likelihood specification. However, in the next section we show that likelihood ratio type tests still produce the same result for all aggregation versions when we assume that the underlying model is correctly specified.

In [21]:
results_all = [res_o, res_f, res_e, res_a]
names = 'res_o res_f res_e res_a'.split()
In [22]:
pd.concat([r.params for r in results_all], axis=1, keys=names)
Out[22]:
res_o res_f res_e res_a
Intercept 2.715533 2.715533 2.715533 2.715533
rate_marriage -0.495180 -0.495180 -0.495180 -0.495180
age -0.029914 -0.029914 -0.029914 -0.029914
yrs_married -0.010763 -0.010763 -0.010763 -0.010763
In [23]:
pd.concat([r.bse for r in results_all], axis=1, keys=names)
Out[23]:
res_o res_f res_e res_a
Intercept 0.107360 0.107360 0.107360 0.107360
rate_marriage 0.011874 0.011874 0.011874 0.011874
age 0.004471 0.004471 0.004471 0.004471
yrs_married 0.004294 0.004294 0.004294 0.004294
In [24]:
pd.concat([r.pvalues for r in results_all], axis=1, keys=names)
Out[24]:
res_o res_f res_e res_a
Intercept 3.756282e-141 3.756280e-141 3.756282e-141 3.756282e-141
rate_marriage 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00
age 2.221918e-11 2.221918e-11 2.221918e-11 2.221918e-11
yrs_married 1.219200e-02 1.219200e-02 1.219200e-02 1.219200e-02
In [25]:
pd.DataFrame(np.column_stack([[r.llf, r.deviance, r.pearson_chi2] for r in results_all]), 
             columns=names, index=['llf', 'deviance', 'pearson chi2'])
Out[25]:
res_o res_f res_e res_a
llf -10350.913296 -10350.913296 -740.748534 -5954.219866
deviance 15374.679054 15374.679054 967.455734 967.455734
pearson chi2 32310.704118 32310.704118 926.199428 926.199428

Likelihood Ratio type tests

We saw above that likelihood and related statistics do not agree between the aggregated and original, individual data. We illustrate in the following that likelihood ratio test and difference in deviance aggree across versions, however Pearson chi-squared does not.

As before: This is not sufficiently clear yet and could change.

As a test case we drop the age variable and compute the likelihood ratio type statistics as difference between reduced or constrained and full or unconstraint model.

original observations and frequency weights

In [26]:
glm = smf.glm('affairs ~ rate_marriage + yrs_married',
              data=data, family=sm.families.Poisson())
res_o2 = glm.fit()
#print(res_f2.summary())
res_o2.pearson_chi2 - res_o.pearson_chi2, res_o2.deviance - res_o.deviance, res_o2.llf - res_o.llf
Out[26]:
(52.91343161871919, 45.726693322507344, -22.863346661251853)
In [27]:
glm = smf.glm('affairs ~ rate_marriage + yrs_married',
              data=dc, family=sm.families.Poisson(), freq_weights=np.asarray(dc['freq']))
res_f2 = glm.fit()
#print(res_f2.summary())
res_f2.pearson_chi2 - res_f.pearson_chi2, res_f2.deviance - res_f.deviance, res_f2.llf - res_f.llf
Out[27]:
(52.91343161873738, 45.726693322507344, -22.863346661253672)

aggregated data: exposure and var_weights

Note: LR test agrees with original observations, pearson_chi2 differs and has the wrong sign.

In [28]:
glm = smf.glm('affairs_sum ~ rate_marriage + yrs_married',
              data=df_a, family=sm.families.Poisson(), exposure=np.asarray(df_a['affairs_count']))
res_e2 = glm.fit()
res_e2.pearson_chi2 - res_e.pearson_chi2, res_e2.deviance - res_e.deviance, res_e2.llf - res_e.llf
Out[28]:
(-31.618527525105605, 45.72669332250632, -22.863346661252876)
In [29]:
glm = smf.glm('affairs_mean ~ rate_marriage + yrs_married',
              data=df_a, family=sm.families.Poisson(), var_weights=np.asarray(df_a['affairs_count']))
res_a2 = glm.fit()
res_a2.pearson_chi2 - res_a.pearson_chi2, res_a2.deviance - res_a.deviance, res_a2.llf - res_a.llf
Out[29]:
(-31.6185275251122, 45.72669332250621, -22.863346661252763)

Investigating Pearson chi-square statistic

First, we do some sanity checks that there are no basic bugs in the computation of pearson_chi2 and resid_pearson.

In [30]:
res_e2.pearson_chi2, res_e.pearson_chi2, (res_e2.resid_pearson**2).sum(), (res_e.resid_pearson**2).sum()
Out[30]:
(894.5809002315146, 926.1994277566203, 894.5809002315145, 926.1994277566203)
In [31]:
res_e._results.resid_response.mean(), res_e.model.family.variance(res_e.mu)[:5], res_e.mu[:5]
Out[31]:
(-1.5172820265154138e-13,
 array([ 5.42753476, 46.42940306, 19.98971769, 38.50138978, 11.18341883]),
 array([ 5.42753476, 46.42940306, 19.98971769, 38.50138978, 11.18341883]))
In [32]:
(res_e._results.resid_response**2 / res_e.model.family.variance(res_e.mu)).sum()
Out[32]:
926.1994277566203
In [33]:
res_e2._results.resid_response.mean(), res_e2.model.family.variance(res_e2.mu)[:5], res_e2.mu[:5]
Out[33]:
(-2.1862853408003084e-15,
 array([ 4.77165474, 44.4026604 , 22.2013302 , 39.14749309, 10.54229538]),
 array([ 4.77165474, 44.4026604 , 22.2013302 , 39.14749309, 10.54229538]))
In [34]:
(res_e2._results.resid_response**2 / res_e2.model.family.variance(res_e2.mu)).sum()
Out[34]:
894.5809002315146
In [35]:
(res_e2._results.resid_response**2).sum(), (res_e._results.resid_response**2).sum()
Out[35]:
(51204.85737832321, 47104.64779595941)

One possible reason for the incorrect sign is that we are subtracting quadratic terms that are divided by different denominators. In some related cases, the recommendation in the literature is to use a common denominator. We can compare pearson chi-squared statistic using the same variance assumption in the full and reduced model.

In this case we obtain the same pearson chi2 scaled difference between reduced and full model across all versions. (Issue #3616 is intended to track this further.)

In [36]:
((res_e2._results.resid_response**2 - res_e._results.resid_response**2) / res_e2.model.family.variance(res_e2.mu)).sum()
Out[36]:
44.433141751219
In [37]:
((res_a2._results.resid_response**2 - res_a._results.resid_response**2) / res_a2.model.family.variance(res_a2.mu)
* res_a2.model.var_weights).sum()
Out[37]:
44.43314175121911
In [38]:
((res_f2._results.resid_response**2 - res_f._results.resid_response**2) / res_f2.model.family.variance(res_f2.mu) 
 * res_f2.model.freq_weights).sum()
Out[38]:
44.43314175121956
In [39]:
((res_o2._results.resid_response**2 - res_o._results.resid_response**2) / res_o2.model.family.variance(res_o2.mu)).sum()
Out[39]:
44.43314175121995

Remainder

The remainder of the notebook just contains some additional checks and can be ignored.

In [40]:
np.exp(res_e2.model.exposure)[:5], np.asarray(df_a['affairs_count'])[:5]
Out[40]:
(array([ 1., 10.,  5., 10.,  3.]), array([ 1, 10,  5, 10,  3]))
In [41]:
res_e2.resid_pearson.sum() - res_e.resid_pearson.sum()
Out[41]:
-9.664817945859852
In [42]:
res_e2.mu[:5]
Out[42]:
array([ 4.77165474, 44.4026604 , 22.2013302 , 39.14749309, 10.54229538])
In [43]:
res_a2.pearson_chi2, res_a.pearson_chi2, res_a2.resid_pearson.sum(), res_a.resid_pearson.sum()
Out[43]:
(894.5809002315142, 926.1994277566264, -42.34720713518869, -32.6823891893241)
In [44]:
((res_a2._results.resid_response**2) / res_a2.model.family.variance(res_a2.mu) * res_a2.model.var_weights).sum()
Out[44]:
894.5809002315142
In [45]:
((res_a._results.resid_response**2) / res_a.model.family.variance(res_a.mu) * res_a.model.var_weights).sum()
Out[45]:
926.1994277566264
In [46]:
((res_a._results.resid_response**2) / res_a.model.family.variance(res_a2.mu) * res_a.model.var_weights).sum()
Out[46]:
850.1477584802954
In [47]:
res_e.model.endog[:5], res_e2.model.endog[:5]
Out[47]:
(array([ 0., 39., 17.,  9.,  4.]), array([ 0., 39., 17.,  9.,  4.]))
In [48]:
res_a.model.endog[:5], res_a2.model.endog[:5]
Out[48]:
(array([0.        , 3.9       , 3.4       , 0.9       , 1.33333333]),
 array([0.        , 3.9       , 3.4       , 0.9       , 1.33333333]))
In [49]:
res_a2.model.endog[:5] * np.exp(res_e2.model.exposure)[:5]
Out[49]:
array([ 0., 39., 17.,  9.,  4.])
In [50]:
res_a2.model.endog[:5] * res_a2.model.var_weights[:5]
Out[50]:
array([ 0., 39., 17.,  9.,  4.])
In [51]:
from scipy import stats
stats.chi2.sf(27.19530754604785, 1), stats.chi2.sf(29.083798806764687, 1)
Out[51]:
(1.8390448369994542e-07, 6.931421143170174e-08)
In [52]:
res_o.pvalues
Out[52]:
Intercept        3.756282e-141
rate_marriage     0.000000e+00
age               2.221918e-11
yrs_married       1.219200e-02
dtype: float64
In [53]:
print(res_e2.summary())
print(res_e.summary())
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:            affairs_sum   No. Observations:                  130
Model:                            GLM   Df Residuals:                      127
Model Family:                 Poisson   Df Model:                            2
Link Function:                    log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -763.61
Date:                Sun, 24 Nov 2019   Deviance:                       1013.2
Time:                        07:50:02   Pearson chi2:                     895.
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
Intercept         2.0754      0.050     41.512      0.000       1.977       2.173
rate_marriage    -0.4947      0.012    -41.743      0.000      -0.518      -0.471
yrs_married      -0.0360      0.002    -17.542      0.000      -0.040      -0.032
=================================================================================
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:            affairs_sum   No. Observations:                  130
Model:                            GLM   Df Residuals:                      126
Model Family:                 Poisson   Df Model:                            3
Link Function:                    log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -740.75
Date:                Sun, 24 Nov 2019   Deviance:                       967.46
Time:                        07:50:02   Pearson chi2:                     926.
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
Intercept         2.7155      0.107     25.294      0.000       2.505       2.926
rate_marriage    -0.4952      0.012    -41.702      0.000      -0.518      -0.472
age              -0.0299      0.004     -6.691      0.000      -0.039      -0.021
yrs_married      -0.0108      0.004     -2.507      0.012      -0.019      -0.002
=================================================================================
In [54]:
print(res_f2.summary())
print(res_f.summary())
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:                affairs   No. Observations:                  476
Model:                            GLM   Df Residuals:                     6363
Model Family:                 Poisson   Df Model:                            2
Link Function:                    log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -10374.
Date:                Sun, 24 Nov 2019   Deviance:                       15420.
Time:                        07:50:02   Pearson chi2:                 3.24e+04
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
Intercept         2.0754      0.050     41.512      0.000       1.977       2.173
rate_marriage    -0.4947      0.012    -41.743      0.000      -0.518      -0.471
yrs_married      -0.0360      0.002    -17.542      0.000      -0.040      -0.032
=================================================================================
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:                affairs   No. Observations:                  476
Model:                            GLM   Df Residuals:                     6362
Model Family:                 Poisson   Df Model:                            3
Link Function:                    log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -10351.
Date:                Sun, 24 Nov 2019   Deviance:                       15375.
Time:                        07:50:02   Pearson chi2:                 3.23e+04
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
Intercept         2.7155      0.107     25.294      0.000       2.505       2.926
rate_marriage    -0.4952      0.012    -41.702      0.000      -0.518      -0.472
age              -0.0299      0.004     -6.691      0.000      -0.039      -0.021
yrs_married      -0.0108      0.004     -2.507      0.012      -0.019      -0.002
=================================================================================