Missing Data Matters (Part II) - Solutions

A Quick Note on MNAR Data

Although we may touch on MNAR data in passing in this second post, the focus will be on MCAR and MAR data. The process for modelling MNAR data is complicated, and I am a massive coward. Cramming the suite of potential MNAR solutions into an already complex post would not do them justice.

However, as a quick summary, there are two main approaches to modelling MNAR data - selection modelling and pattern mixture modelling. Selection models treat missingness as the outcome in a regression, similar to a more precise version of the logistic regression fit in the previous post, while pattern mixture models treat the missingness as an explanatory variable.

For more information about MNAR modelling processes, I’d recommend Enders (2022)’s Applied Missing Data Analysis.

And Another on Likelihood Methods for Good Measure

One group of methods I haven’t discussed here is likelihood methods, such as full information maximum likelihood (FIML). In simple terms, likelihood methods borrow information from complete cases to improve the estimation of parameters with missing data (Peters and Enders 2002). Maximum likelihood uses the available information to estimate a log-likelihood function most likely to have produced the observed data (Enders 2022). In particular, FIML considers only complete cases when calculating the log-likelihood function and maximises it using an expectation-maximisation algorithm (Xu, Chen, and Li 2022).

However, implementing methods like FIML in R is a little tricky³, so these methods are out of scope for my rubbish little blog post. If you want to learn more about the implementation of FIML in R, I’d recommend the “Full Information Maximum Likelihood” section of William Murrah’s Statistical Thinking site and there the lavaan package documentation can also help figure out how to compute FIML to deal with missing data. If I am feeling like a total mad lad later on, I may write another post about implementing FIML.

Single Imputation

There are a few different forms of single imputation, the most common being simple imputation methods that replace all missing values with the variable’s average value (usually the mean, but median and mode can be used too) or replace missing values using randomly drawn values from the same variable, otherwise known as simple hot-deck imputation (Xu, Chen, and Li 2022). Simple imputation methods are almost always harmful, especially methods like mean imputation. They distort the variable’s distribution, underestimating its variance (Van Buuren 2018), disrupting the relationship between the variable with imputed values and all other variables (Nguyen 2020), and biasing model estimates (Van Buuren 2018). It is important to know that these simple imputation methods exist, but the circumstances where using them is valid are very few!

Another approach is to estimate a regression model that predicts missing values. Regression imputation could take the form of any regression model, depending on the data type and complexity, with linear and logistic regression being the most common. However, Van Buuren (2018) argues that regression imputation is the “most dangerous” method because it gives false confidence in outputs.

We can look at the effects of imputation by visualising imputation methods. I have transformed the total working years value for 75% of employees with high or very high relationship satisfaction to NA. Obviously, this is a very high proportion of missing values. If 75% of your data is missing, you should be deeply alarmed, stop whatever you are doing, and bin that dataset. However, in this case, it is meant to illustrate how imputation methods work, and having a disproportionately large number of missing values makes it easier to see the effects visually.

set.seed(123)

missing_years <-
  attrition |> 
  mutate(
    total_working_years = replace(
      total_working_years, runif(n()) < 0.75 & 
        relationship_satisfaction %in% c("High", "Very High") , NA
    ),
    missingness = ifelse(is.na(total_working_years), "Imputed", "Observed")
  )

plot_regression <-
  function(data) {
    if ("missingness" %in% names(data)) {
      data |>
        ggplot(aes(x = total_working_years, y = monthly_income)) +
        geom_point(aes(colour = missingness), size = 1.5, alpha = .6) +
        geom_smooth(method = lm, colour = "#026E99", fill = "#026E99", alpha = .6) +
        scale_colour_manual(values = c("#D93649", "#343A40")) + 
        scale_y_continuous(labels = scales::label_currency()) +
        guides(colour = guide_legend(override.aes = list(size = 3))) +
        labs(
          x = "Total Working Years", y = "Monthly Income",
          caption = "Visualisation: Paul Johnson  |  Data: IBM Watson Analytics Lab"
        )
    } else {
      data |>
        ggplot(aes(x = total_working_years, y = monthly_income)) +
        geom_point(colour = "#343A40", size = 1.5, alpha = .6) +
        geom_smooth(
          method = lm, colour = "#026E99",
          fill = "#026E99", alpha = .6
        ) +
        scale_y_continuous(labels = scales::label_currency()) +
        labs(
          x = "Total Working Years", y = "Monthly Income",
          caption = "Visualisation: Paul Johnson  |  Data: IBM Watson Analytics Lab"
        )
    }
  }

attrition |>
  plot_regression() +
  labs(
    title = "No Missing Data & No Imputation",
    subtitle = 
      stringr::str_wrap(
        glue::glue(
          "Employee's monthly income by total length of their career, without ",
          "missing values and without imputation."
          ),
        width = 90
      )
    )

missing_years |>
  mice::mice(
    method = "mean", m = 1,
    maxit = 1, print = FALSE
  ) |>
  mice::complete() |>
  plot_regression() +
  labs(
    title = "Mean Imputation",
    subtitle = 
      stringr::str_wrap(
        glue::glue(
          "Employee's monthly income by total length of their career, with ",
          "missing values replaced using mean imputation."
        ),
        width = 90
      )
    )

missing_years |> 
  mice::mice(
    method = "norm.predict", m = 1,
    maxit = 1, print = FALSE
  ) |>
  mice::complete() |>
  plot_regression() +
  labs(
    title = "Linear Regression Imputation",
    subtitle = 
      stringr::str_wrap(
        glue::glue(
          "Employee's monthly income by total length of their career, with ",
          "missing values replaced using regression imputation with linear ",
          "regression."
        ),
        width = 90
      )
    )

Imputing missing values using simple mean imputation is quick and easy, but the results clearly leave much to be desired. However, estimates can sometimes be useful even if they are imprecise. That’s not the case here either because mean imputation narrows the distribution of total working years. The confidence intervals around the regression line are narrower around the mean but significantly wider at the higher end of total working years.

The regression imputation is okay at first inspection, but it imputes values of total working years to six decimal points while the actual values are all integers. The imputation also thinks that some people have worked negative years! These issues can be fixed by giving mice the necessary details to impute more realistic values. However, the razor-thin error bars, particularly at the lower values of total working years, cannot be fixed using single imputation.

This points to the inherent flaw with single imputation that none of these methods can escape. In Rubin (1987)’s words:

Imputing one value for a missing datum cannot be correct in general, because we don’t know what value to impute with certainty (if we did, it wouldn’t be missing).

We cannot account for the uncertainty in imputing missing values using single imputation because we are plugging in a single value in place of the missing value and treating it with the same confidence as the observed values. There are ways to add more uncertainty to single imputation methods, such as stochastic regression imputation, but ultimately, these are papering over the cracks. Single imputation always offers a false sense of certainty because it imputes a single value in place of each missing value.

Multiple Imputation

Multiple imputation attempts to address the fundamental flaw in single imputation methods, acknowledging the uncertainty in imputed values and baking that uncertainty into the process. Multiple imputation involves generating multiple datasets, performing analysis on each, and pooling the results. The multiple imputation process yields parameter estimates that “average over a number of plausible replacement values” (Enders 2022, 189) instead of the one that single imputation does. Multiple imputation can be appropriate for missing data that is not MAR, as long as the response mechanism has been explicitly modelled or other identifying assumptions have been made (Murray 2018).

Enders (2022) splits the multiple imputation process into three phases - imputation, analysis, and pooling.

1. Imputation Phase - generates multiple, \(m\), complete datasets, estimating the missing values plus a random component to capture the uncertainty in the estimate,
2. Analysis Phase - computes parameter estimates using typical statistical methods, but computed \(m\) times and applied to the \(m\) completed datasets,
3. Pooling Phase - combines these estimates⁷ and calculates the pooled estimates and their standard errors.

While there are multiple ways to approach each phase, the three phases are typical across multiple imputation.

Here, we will use multiple imputation by chained equations (MICE). MICE imputes one variable at a time, using a separate model for each incomplete variable, a process called Fully Conditional Specification. Plausible values are generated using the rest of the data columns (using various methods depending on the variable) in an iterative process that uses Gibbs sampling (Van Buuren and Groothuis-Oudshoorn 2011).

Effective multiple imputation requires as much information as possible, including variables associated with the missingness itself and with the observed values of that variable (which are, therefore, likely to be associated with the missing values). Thus, Murray (2018) argues that imputation models should include as many variables as possible.

The plot below shows the results of the imputation phase of the multiple imputation process, with the pooling carried out manually, but it is important to note that the results are not a perfect implementation of multiple imputation. Instead, each of the five completed datasets has been stacked together, and the mean value of each row⁸ across those five datasets has been calculated. The results plotted below do not perfectly demonstrate what multiple imputation looks like, but they are at least an approximation.

missing_years |> 
  mice::mice(method = "pmm", print = FALSE, seed = 123) |>
  mice::complete(action = "long") |> 
  summarise(
    across(c(total_working_years, monthly_income), ~ round(mean(.x), 0)), 
    .by = c(.id, missingness)
    ) |> 
  plot_regression() +
  labs(
    title = "Multiple Imputation with Predictive Mean Matching",
    subtitle =
      stringr::str_wrap(
        glue::glue(
          "Employee's monthly income by total length of their career, with missing ",
          "values replaced using multiple imputation with predictive mean matching."
        ),
        width = 90
      )
    )

The confidence intervals don’t capture the baked-in uncertainty, a significant advantage of multiple imputation. The plot only shows the final mean values calculated from the five imputed datasets. However, the multiple imputation process results follow a similar distribution to the original data, which should reduce the risk of imputations producing biased parameter estimates. Further, the imputed values appear to be structured similarly to the observed data, making the multiple imputation plot look very similar to the original data plot. This is generally a good sign because the closer you are to replicating the data-generating process, the better your chance of making accurate inferences. The structural similarities are primarily due to the method used to carry out the multiple imputation, predictive mean matching.