Ordered Categorical Variables

Recall partyid3. Here, ‘1’ represents a preference for the Democratic candidate; ‘2’, no preference; and ‘3’, a preference for the Republican candidate. This is an ordered categorical variable, and it is natural to prefer a design matrix that maps these values to -1, 0, and 1 respectively. We might accomplish this as follows:

kw = {
    'dtype' : {
        'gender' : 'category',
        'partyid3' : pd.CategoricalDtype([1,2,3], ordered=True),
        'race' : 'category'
    }
}
df = read_csv( **kw )

df['partyid3'] = df.partyid3.map({1:-1,2:0,3:1})

# an ordered categorical admits min/max
(df.partyid3.min(), df.partyid3.max())

(-1, 1)

This feels ugly. In particular, this requires a single idea–coding the categorical varible–to be separated into two distinct steps occuring in separate places in the code. Furthermore, there are now three representations to consider, and we’ve discarded the original coding. This will make it more difficult to cross check against the original data.

Reference categories

For a popular class of simple regressions models, namely linear and generalized linear regressions, it is crucial that the design matrix be full rank. This requires constraints on how categorical variables are coded. Usually, this is handled by introducing reference categories.

For example, in their 2007 textbook, Data Analysis Using Regression and Multilevel/Hierarchical Models, Gelman and Hill define the reference category to be white and male and introduce two boolean variables, one for female and one for black. In the context of the fitted model, the coefficients associated with the two boolean variables are interpreted as level shifts from the white and male reference group. Social scientists often define reference groups based on the specific questions to be addressed by the study.

We can encode this as follows:

kw = {
    'dtype' : {
        'gender' : pd.CategoricalDtype([1,2]),
        'partyid3' : pd.CategoricalDtype([1,2,3], ordered=True),
        'race' : pd.CategoricalDtype([1,2,3])
    }
}
df = read_csv( **kw )

# map: internal codes -> gender
gender_map = dict( enumerate( df.gender.cat.categories ) )

# map: internal codes -> race
race_map = dict( enumerate( df.gender.cat.categories ) )

# reference levels are at zero index
[gender_map[0], race_map[0]]

[1, 1]

New features in ExtCategorical

Let’s investigate the new features. The main utility comes from the following accessor, useful during adhoc exploratory analysis or constructing design matrices.

First, an ext_category looks and feels like a normal categorical variable:

# the usual display
df.partyid3

record_id
19641    1
19642    3
19643    3
19644    3
19645    2
Name: partyid3, dtype: ext_category

But we can ask it to masquerade as though the metadata was explicitly available:

# act like a categorical with catagories that match the metadata
partyid3_metadata = df.partyid3.xcat.relevel(1)
partyid3_metadata

record_id
19641    prefers Democratic party
19642    prefers Republican party
19643    prefers Republican party
19644    prefers Republican party
19645               no preference
dtype: ext_category

or, as a categorical variable that uses the design matrix codings

# act like a categorical with catagories that match the design matrix codings
partyid3_design = df.partyid3.xcat.relevel(2)
partyid3_design

record_id
19641   -1
19642    1
19643    1
19644    1
19645    0
dtype: ext_category

Or convert back to the values found in the original data:

partyid3_raw = partyid3_design.xcat.relevel(0)
partyid3_raw

record_id
19641    1
19642    3
19643    3
19644    3
19645    2
dtype: ext_category

The argument to relevel is zero indexed and corresponds to the order in which we defined the ExtCategoricalDtype in the dictionary passed to pandas read_csv function.

OOP Design

To downstream libraries that anticipate categorical variables, the ExtCategoricalDtype enjoys the following property: it’s a subclass of CategoricalDtype, and instances of ExtCategoricalDtype behave accordingly:

isinstance(partyid3_design.values, pd.Categorical)

True

This means that well written packages should be able to handle an ExtCategorical transparently, exactly as they would handle a Categorical dtype.

Additional Functionality

To date, we’ve implemented basic functionality. Methods like dropna should just work as expected:

display(df)
df.dropna()

And we can add and replace columns in a data frame:

df['race_m'] = df.race.xcat.relevel(1)
df['partyid3_coded'] = df.partyid3.xcat.relevel(2).astype(int)
df

Final words

We’ve only handled a few use cases; I’m sure there are plenty of code paths that we haven’t walked down. That means your mileage may vary. However, we’re interested in improving this as time permits, so be in touch if there are bugs or feature requests. Thanks!