Transcript for 2026 Summer Training Series Session 4: Handling Missing Data Presenter: Alexander F. Roehrkasse, Ph.D., Butler University National Data Archive on Child Abuse and Neglect (NDACAN) [MUSIC] [VOICEOVER] National Data Archive on Child Abuse and Neglect. [ONSCREEN CONTENT SLIDE 1] Welcome to the 2026 NDACAN Summer training series! The session will begin at 12pm EST. This session is being recorded. Please submit questions to the Q&A box! [Alyssa Lindsey] Hello, everyone! Welcome to our 2026 NDACAN Summer Training Series. As folks begin to join, I'm just going to go ahead and get started with some housekeeping items. My name is Alyssa Lindsey. I'm the Graduate Research Associate here at NDACAN. Before we get started, just want to note a few things. We are recording this session, so we can post the recording, the transcripts, slides, any other materials on our website several weeks following each presentation, and I'll announce their availability on the Child Maltreatment Research L, or CMRL, listserv. And I'll put more information on how to join that listserv in the chat after I pass it off. Please let me know if you have any questions by using the Q&A box, and you can use that Q&A box throughout the presentation, and we'll address all of those questions at the end. That Q&A box can be found in the lower right-hand corner of your Zoom screen. Next slide. [ONSCREEN CONTENT SLIDE 2] NDACAN Summer Training series. National Data Archive on Child Abuse and Neglect Duke University, Cornell University, UC San Francisco, & Mathematica [Alyssa Lindsey] Great. So, welcome again to the NDACAN Summer Training Series. NDACAN stands for the National Data Archive on Child Abuse and Neglect, and it's housed at Duke University, Cornell University, UC San Francisco, and Mathematica. Next slide. [ONSCREEN CONTENT SLIDE 3] Laying the groundwork foundational skills for using administrative data in child welfare research. Logo for the Children's Bureau features an image of overlapping blue and white silhouettes of children next to red and white stripes. To the right is the text "Children's Bureau: An Office of the Administration for Children & Families." Logo image which features a semicircle of icons representing people holding hands positioned on top of the acronym NDACAN, and next to the text National Data Archive Child Abuse and Neglect. [Alyssa Lindsey] NDACAN does two learning offerings every year. We have our monthly office hours series during the academic year, and in the summertime, we have the summer training series. The theme of this year, 2026, Summer Training Series is Laying the Groundwork, Foundational Skills for Using Administrative Data in Child Welfare Research, and this series is designed to strengthen foundational skills, to conduct high-quality, impactful research with administrative data. And we're intending this to be useful both for folks who are newer to this area, and also for those who are seeking, more of an intense or, specific experience. Next slide. [ONSCREEN CONTENT SLIDE 4] NDACAN Summer Training series schedule. July 1st: Overview of NDACAN administrative datasets July 8th: Data cleaning and management July 15th: Linking NCANDS and AFCARS July 22nd: Handling missing data July 29th: Data presentation and visualization [Alyssa Lindsey] And this is a preview of the rest of our schedule for the 2026 Summer Training Series. We'll be meeting every Wednesday at 12 Eastern for an hour in July. So far, in the summer training series, we've presented an overview of NDACAN administrative data sets, went over data cleaning and management, and linking NCANDS and AFCARS. The materials from the first few sessions are currently available on the NDACAN website, and subsequent sessions, like I mentioned, will be available and announced on the listserv shortly. I'll provide links for where to find those sessions in the chat after I pass it off. And today's session will cover handling missing data. Next week's final session of the 2026 Summer Training Series will cover data presentation and visualization. Next slide. [ONSCREEN CONTENT SLIDE 5] Session Agenda. Handling missing data. Demonstration in R. Q & A. [Alyssa Lindsey] And, as I mentioned, today's session is going to start with a presentation on handling missing data, and then a demonstration in R, followed by Q&A. Next. [ONSCREEN CONTENT SLIDE 6] Handling missing data. [Alyssa Lindsey] Great, now I'm going to pass it over to Alex to lead the rest of today's discussion. Thank you. [Alex Roehrkasse] Thanks, Alyssa. My name is Alex Roehrkasse, I'm an assistant professor of sociology and criminology at Butler University, and also a research associate at the National Data Archive on Child Abuse and Neglect. I'm really excited to be here with you this afternoon. I gave the last couple presentations on data cleaning and data linking. Today's presentation is going to build a little on those presentations, but if you weren't at those presentations, don't worry, this is a standalone presentation. Also, as with the previous presentations, missing data is a really big topic. We're just going to be skimming the surface today. We're going to be talking about some foundational skills in general for handling missing data, but also be talking about some tips, tricks, best practices for working with missing data in the specific administrative data sets that we distribute through the archive. A reminder, just as Alyssa said, that everything I'm presenting today is going to be published on our website, so if you're having a hard time following along, don't stress, you can find the slides, the code, a video recording of the presentation, shortly. Alright, let's get started. [ONSCREEN CONTENT SLIDE 7] Types of missing administrative data. Record-level missingness I.e. observations that should appear in your dataset don’t. Item-level missingness I.e. values that should appear in your dataset don’t. [Alex Roehrkasse] I want to start today's presentation by outlining what I understand to be two of the most important distinctions in terms of types of missing data that arise in the administrative data that we distribute. One is more commonly acknowledged, and there's a much larger literature about how to handle it. The other is less frequently understood, and it's not as obvious how to handle this case. The less frequently understood, sometimes trickier case, is the case of record-level missingness. This is a scenario where whole observations, or whole rows of data that should appear in your dataset don't. This is in contrast to what we call item-level missingness, which is a scenario where you do observe all of the necessary records that you'd like to observe, you have all the rows in your dataset that you need, it's just that values that should appear in your dataset don't. Item level missingness will appear as empty cells or blank cells in your data frame. Record level missingness is harder to identify because it's the simple absence of rows in your dataset. [ONSCREEN CONTENT SLIDE 8] Record-level missingness in nDACAn administrative data. Some NDACAN datasets have meaningful amounts of missing observations. E.g. 100% of maltreatment reports in NCANDS are missing in Arizona in 2021. Not all record-level missingness is accounted for in User Guides. [Alex Roehrkasse] Okay, with this distinction in mind, let's talk about each of these kinds of missingness, specifically in the administrative data sets that we distribute. Let's tackle record-level missingness first. Some archived datasets have meaningful amounts of missing observations. What's an example of this? Well, 100% of maltreatment reports are missing in NCANDS from Arizona in 2021. Not all record-level missingness is completely accounted for in NDACAN user guides, and so it's extremely important that you investigate record-level missingness on your own, using the data that you plan to analyze. [ONSCREEN CONTENT SLIDE 9] Sources of Record-level missingness. State-level non-reporting. Delayed reporting. Data suppression (e.g. tribal agencies). [Alex Roehrkasse] Why are there record-level missingness in archive data? Most often, it's owing to state-level non-reporting of certain data. There are just certain states in certain years that didn't report information to the Children's Bureau, and so those records are missing in the datasets that we distribute. Sometimes, record-level missingness can result from delayed reporting. The data are there, they just don't show up until some file further down the line. We've talked about this problem, and I'll demonstrate it in our demonstration in R later in today's presentation. Lastly, another source of record-level missingness is data suppression. The most significant, frequently overlooked example of this is, records of maltreatment from Child Protective Services operated by tribal agencies. So, tribal CPS agencies, most of them at least, report to the Children's Bureau, but the Children's Bureau suppresses those records in the research files that we distribute. Another example of record-level missingness. [ONSCREEN CONTENT SLIDE 10] State-level nonreporting. Table summarizing missing data for the NCANDS Child Files between 2003–2022. Source: NDACAN internal documentation for NCANDS Child Files 2007–2022. [Alex Roehrkasse] Let me illustrate the problem of record-level missingness. Here is the case of state-level non-reporting. This table is from internal documentation from archive staff, where we tabulate the number of states that reported in the NCANDS child files over the last 15 or so years. What you can see is that in most recent years, all 50 states, Puerto Rico, and the District of Columbia report NCANDS data. But occasionally, a state doesn't report. There you see, in 2021, Arizona failed to report any records to NCANDS. Ditto Puerto Rico in 2016. Before 2012, this becomes an even bigger problem, when a larger number of states were not regularly reporting to NCANDS. For this reason, I encourage people working with NCANDS before 2012 to think especially carefully about record-level missingness. [ONSCREEN CONTENT SLIDE 11] Delayed reporting. A bar chart in showing the number of records observed in a sample of NCANDS data every half-month in fiscal year 2019. In most half-months, most records come from the 2019 NCANDS Child File, but toward the end of the year, a meaningful proportion come from the 2020 NCANDS Child File. Note: Results based on 1% random samples of the 2019 and 2020 NCANDS Child Files. [Alex Roehrkasse] Here's an illustration of the delayed reporting problem. You've seen a version of this slide in previous presentations this summer. Briefly, let's say you wanted to count up all of the maltreatment reports that occurred in the 2019 federal fiscal year. If you just used the 2019 NCANDS child file, you'd notice that reports started to taper off at the end of the federal fiscal year. This is owing almost entirely to the fact that those reports didn't make it into the 2019 child file. They made it only into the 2020 child file. Those records show up as the red areas in this bar chart. So without careful thinking about combining different data sources, we might have record-level missingness owing to delayed reporting. [ONSCREEN CONTENT SLIDE 12] Item-level Missingness in NDACAN administrative data. Most NDACAN datasets have meaningful amounts of missing values for variables of common interest. E.g. 63% of maltreatment reports in NCANDS in 2019 have missing values for “Financial Problems.” Not all missing values follow the formatting described in Codebooks. [Alex Roehrkasse] Okay, what about item-level missingness in archived administrative data? Most administrative data sets also have meaningful amounts of missing values for variables of common interest. Just a random example. About 63% of maltreatment reports from 2019 have missing values for a variable capturing a certain child risk factor, namely whether or not their household experienced financial problems. Not all missing values follow the formatting described in the codebooks we distribute. This is a problem we talked about in our data cleaning and management session. And for this reason, it's really, really important for you to independently and manually verify the encoding of all variables that you plan on using. [ONSCREEN CONTENT SLIDE 13] Sources of item-level missingness. State- and agency-level processes. Caseworker-, caregiver-, and child-level processes. Data suppression (e.g. geographic identifiers). Linkage failure. [Alex Roehrkasse] Why is there item level missingness in archive administrative data? Again, sometimes this results from state or county-level agency processes, where a state or an agency simply doesn't, as a matter of practice or procedure, collect that information. So it'll be systematically missing from some reporting unit. Sometimes, though, a specific caseworker or caregiver or child will cause data to be missing. Maybe some specific caseworker fails to record it. Or it's difficult to ascertain some information about a specific child. Data suppression also can lead to item-level missingness. The most common example here is where children coming from small counties with a small number of reports will have their county-level geographic identifiers suppressed in the research files we distribute. Lastly, harkening back to last week's presentation on data linkage. Whenever we have failed linkages in our linked data. This can yield missing values for variables coming from one of the two data sources. [ONSCREEN CONTENT SLIDE 14] Identifying missing data. NDACAN staff can provide, upon request, internal documentation of record-level missingness. But the bread-and-butter techniques for identifying missing data are data cleaning and exploratory analysis. [Alex Roehrkasse] How do we find missing data? How do we know that we have missing data, and how much missing data we have? Archive staff can provide, upon request, some documentation, particularly about record-level missingness, but the bread-and-butter techniques for identifying missing data are data cleaning and exploratory analysis. What does that look like? [ONSCREEN CONTENT SLIDE 15] Data cleaning for identifying missing data. R may not understand correctly coded missing values as missing values. Missing values may not be correctly or consistently coded. Identifying all possible encodings of missing values is essential. E.g. “”, “99,” “-1,” “Don’t know,” or “Decline to respond.” Encode missing values so that R understands them as such (NA). If retaining available information about the missing-data mechanism is important, consider “tagging” NA values using packages like haven. [Alex Roehrkasse] This slide harkens back to session 2 of this summer's Summer Training Series. So, a brief recap. It's important to understand that R may not understand missing values that are encoded in the way the codebook says they should be. Many missing values, according to the codebook, should be coded as 99, or negative 1. R will not understand these as missing values. Secondly, missing values may not be correctly or consistently encoded. For this reason, it's important to identify all the different ways in which missing values might be encoded in your dataset and then to re-encode them, or to clean them, so that R understands them as missing values. Generally speaking, R understands missing data as NA. By default, NA is NA is NA. Missing is missing is missing. Sometimes, though, we want to retain information about why the data is missing, and so it is possible to tag NA values, designating certain kinds of NA values corresponding to certain reasons why the data might be missing. [ONSCREEN CONTENT SLIDE 16] Exploratory analysis for identifying missing data. Tabulate or visualize the frequency distribution of all variables. Stratify by geographic unit of analysis. When possible, visualize trends and trend breaks in all variables. [Alex Roehrkasse] What does exploratory analysis do for the identification of missing data? I often recommend that users tabulate or visualize the frequency distribution of all variables that they plan to work with. It's especially useful to do this stratifying by geographic unit of analysis, usually the state, but sometimes the county. When possible, it's almost always helpful to visualize trends and trend breaks in all variables. This is especially helpful when missing data aren't encoded correctly. [ONSCREEN CONTENT SLIDE 17] Handling Record-level missingness. Limiting your sampling frame so that records are missing in a consistent way can be a helpful way of avoiding threats to internal validity. If properly accounted for, missing observations are therefore primarily a threat to external validity. Missing-data strategies can be applied to record-level missingness, but are not as common. Therefore, transparency about sampling and inference is often most important. [Alex Roehrkasse] Okay, once we've identified missing data, what do we do about it? Let's take first the case of record-level missingness. When we have record-level missingness in our data, often limiting our sampling frame so that records are missing in a consistent way can be a helpful way of avoiding threats to internal validity, making sure that our results aren't biased. If properly accounted for, missing observations then are mostly a threat to external validity, the generalizability of our inferences. The missing data strategies that we'll talk about when we talk about item-level missingness can be applied to record-level missingness in special cases, but they're not as common. Therefore, often, once we've understood our record-level missingness and sampled appropriately. Transparency and, honestly, humility about inference is often most important. [ONSCREEN CONTENT SLIDE 18] Handling item-level missingness. A wide variety of approaches to handling missing values are feasible with NDACAN administrative data: Complete-case analysis. Hot-deck imputation. Multiple imputation. Full-information maximum-likelihood. Bayesian methods. Additional data collection. [Alex Roehrkasse] What about item-level missingness? This is a scenario where there's considerably more resources available to us, a much larger literature, a much larger menu of techniques that we can use to address the missing data problem. A wide variety of approaches to handling missing values are feasible with the administrative data that we distribute. This is just a very short list, but a list that captures some of the most common ways that users handle missing data in our datasets. I'll walk through them very briefly. Complete case analysis, also sometimes called list-wise deletion is the scenario where you simply drop from analysis any observations that have missing values for variables that you're analyzing. Hot deck imputation is a mostly outdated approach that borrows information from similar observations and simply carries it forward into observations with missing values. Even though it's mostly outmoded, I mention it because there are certain properties of archive administrative data that actually make hotdeck imputation pretty helpful for dealing with certain kinds of missing data. I'll discuss that in the demonstration. The sort of bread-and-butter method for dealing with item-level miscellaneous these days is called multiple imputation. This is a model-based approach, where we use a statistical model to make inferences about missing values using observed information. We do this multiple times, generating essentially multiple copies of our data. And then we use variants across these imputations, or copies of our dataset, to communicate uncertainty about our guesses about the true values of missing values. Other options, like maximum likelihood and Bayesian methods, are becoming increasingly popular. They're a little bit more, advanced, and therefore beyond the scope of today's presentation, but we're always happy to talk about use cases and applications. Lastly, sometimes, it's simply the case that additional data is necessary to make valid inferences about missing data. And I'll talk briefly about that as well. [ONSCREEN CONTENT SLIDE 19] The stakes of handling missing data appropriately. Without an appropriate strategy, inferences from data with missing values and/or missing observations may be biased and/or inefficient. The appropriate missing data strategy depends on both the missing-data mechanism and your analytic goals. [Alex Roehrkasse] Candidly, many people don't think very hard about missing data. Some people don't think about it at all. What are the stakes? What do we gain by thinking about it carefully? What do we lose by not handling data, missing data appropriately? Without an appropriate missing data strategy, inferences that we make based on data with missing values or missing observations can be biased. That is to say, our estimate is too high or too low. Or they can be inefficient, which often means that they're imprecise. We have less precision in our estimate than we might otherwise. The appropriate missing data strategy often depends on both the underlying missing data mechanism, and your analytic goals. I want to walk through how each of those, the missing data mechanism and the analytic goal, may influence the best choice for handling missing data. [ONSCREEN CONTENT SLIDE 20] Missing-Data mechanisms. Missing completely at random (MCAR). Missing at random (MAR). Missing not at random (MNAR). [Alex Roehrkasse] Okay, when I talk about missing data mechanisms, what I mean is, essentially, the process that caused your value to be missing. What is the process that led your value to be missing as opposed to observed? Broadly, we categorize missing data mechanisms into three different families. Missing completely at random, missing at random, and missing not at random. I'll walk through each of these, explaining what they mean, giving an example of each, and what kinds of strategies are feasible according to each missing data mechanism. [ONSCREEN CONTENT SLIDE 21] Missing Completely at Random (MCAR). The missing-data mechanism is unrelated to the data-generating process. In other words: missingness is not predicted by observed or missing values. E.g. when they haven’t had enough coffee, caseworkers sometimes forget to record information about children’s emotional disturbance. Candidate solutions: complete-case analysis, hot-deck imputation, multiple imputation, full-information maximum-likelihood, Bayesian methods, additional data collection. [Alex Roehrkasse] The first scenario is where our data are missing completely at random. If our data are missing completely at random, it means that the process causing missingness, the missing data mechanism, is unrelated to the data generating process. Or the process that causes certain true values to arise. Otherwise stated, missingness in this case is not predicted by observed or missing values. Let me give an example. Say that when caseworkers haven't had enough coffee, sometimes they forget to record information about children's emotional disturbance, another risk factor that's supposed to be recorded in NCANDS. In this case, whether or not children's emotional disturbance is missing or not just relates to how much coffee the caseworker drank, which we can treat as, like, pretty random or exogenous. It doesn't depend on the child's actual, true level of emotional disturbance. In this case, a variety of solutions are feasible, and the best one to choose depends on, other considerations, including your analytic goals. Unfortunately, it's rarely the case that data are completely randomly missing. [ONSCREEN CONTENT SLIDE 22] Missing at Random (MAR). The missing-data mechanism is related to the data-generating process in observable ways. In other words: missingness is predicted only by observed values. E.g. states vary in their procedures for recording children’s emotional disturbance. Candidate solutions: (not complete-case analysis), hot-deck imputation, multiple imputation, full-information maximum-likelihood, Bayesian methods, additional data collection. [Alex Roehrkasse] Much more common is the scenario where our data are missing at random. What does this mean? If our data are missing at random, the missing data mechanism is relating to the data generating process, but only in observable ways. In other words, missingness is predicted only by observed values. Let me give an example of this, probably the most common. States vary in the ways their procedures for recording children's emotional disturbance. In other words. Some children have missing values for emotional disturbance because they live in states that simply don't record that information. Other states have observed values for emotional disturbance, because they live in states where there's policies and procedures that dictate that that information will be recorded. We can use the state from which a report comes, which is observable, to predict whether the observation will be missing or not. Here, complete case analysis will not yield valid inferences, but most other strategies will work if appropriately implemented. [ONSCREEN CONTENT SLIDE 23] Missing NOT at Random (MNAR). The missing-data mechanism is related to the data-generating process in unobservable ways. In other words: missingness is predicted by the missing values themselves. E.g. caseworkers are less likely to record information about children’s emotional disturbance when they do not experience emotional disturbance. Candidate solutions: (not complete-case analysis, not hot-deck imputation, not multiple imputation, not full-information maximum-likelihood, not Bayesian methods) additional data collection. [Alex Roehrkasse] Lastly, a much more pernicious scenario is where our data are missing not at random and here, the missing data mechanism is related to the data generating process in unobservable ways. The most common example here is where missingness is predicted by the missing values themselves. An example. Caseworkers are less likely to record information about children's emotional disturbance when those children do not, in fact, experience emotional disturbance. This is actually pretty plausible. Let's say if the caseworker does not observe emotional disturbance in a child, they simply don't write anything down. They only record information about emotional disturbance if they observe emotional disturbance. In this case, the true value of the missing variable predicts the missingness. Unfortunately, in this scenario, most strategies for handling missing data aren't feasible, and we very often have to go back to the drawing board, draw on additional information. To solve this problem. [ONSCREEN CONTENT SLIDE 24] Patterns of item-level missingness. Table summarizing missing data for the 2021 NCANDS Child File. The variables are CdAlc , CdDrug , CdEmotnl , CdVisual , CdLearn , CdPhys , CdBehav , CdMedicl and the states are Alabama, Arkansas, California, Colorado, Connecticut, Delaware, District of Columbia, Florida, Georgia, Hawaii, Idaho, Illinois. Note: Numbers are percentages of records with non-missing values in the 2021 NCANDS Child File. [Alex Roehrkasse] I said that it's important to understand the missing data mechanism. But we really can't observe the missing data mechanism directly. We have to make inferences about it by examining patterns of missingness in our data. Let me illustrate how you might go about doing that. Here you see in the first column a few different states, and then across the different columns, a number of different measures from the NCANDS capturing children's risk factors. Child alcohol use, child drug use, child emotional disturbance, child visual impairment, etc, etc. Each cell, then, represents, for the 2021 child file, the proportion of observations That have non-missing values for each variable. You can see that in Alabama, we have complete data. We observe all of these variables for every child. In Idaho, we have none of this information. We collect none of these variables for any child from that state. Many other states have that information for some, but not all children, and seem to have a very stable proportion of children with information about each variable. We don't know definitively what the missing data mechanism is here, but there are some suggestive patterns. It doesn't seem to be the case that our data are missing completely at random. Clearly, they vary systematically by state. But neither does it seem to be the case that our data are missing not at random. Things would be a little noisier if they were. I look at this table, and I see suggestive evidence that our data are mostly missing at random. Specifically, missing at random, conditional on the state from which the data come. What that means is that we can leverage information about the geographic origins of these records to make pretty strong inferences, About what the true values might be. [ONSCREEN CONTENT SLIDE 25] Your analytic goals. Is your estimand a population parameter such as a count or an incidence rate? Is your estimand a model parameter such as a causal effect? Is your estimand a prediction or classification? [Alex Roehrkasse] Lastly, I said that your analytic goals should also guide your choice of missing data strategy. I won't say too much about this, because it gets pretty complicated, and it's really about specifically tailoring to your specific research question. But broadly speaking, you want to ask yourself. What is your estimate? What is the quantity that you are ultimately trying to calculate? Is this estimand a population parameter, like a count or an incidence rate? Is this estimand a model parameter, like a causal effect? Is your estimand a prediction, or some sort of classification? The answer to questions like this should guide which missing data strategy you ultimately end up employing. [ONSCREEN CONTENT SLIDE 26] Best practices for handling missing data. Define an estimand clearly. Use available data to make inferences about missing-data mechanisms. Choose an appropriate, primary missing-data strategy. Conduct sensitivity analysis using reasonable alternative strategies. Report information about data missingness, key assumptions, diagnostics, and sensitivity analysis. [Alex Roehrkasse] I want to start to bring this part of the presentation to a close, leaving you with a sort of sequence of best practices for handling missing data. First, it's imperative to define your estimate clearly early on in planning a missing data strategy. What is it, ultimately, that you're trying to estimate? Then, use available data to make inferences about missing data mechanisms. Why are your data missing? What possible process causes your data to be missing? According to your estimand and your missing data mechanism, then choose an appropriate primary missing data strategy. But then, conduct sensitivity analyses. Try some alternative, reasonable strategies for missing data, for handling missing data. And examine whether or not reasonable alternative strategies yield consistent or highly varying results. Lastly, always report information about the extent and patterns of missingness in your data. Any key assumptions you make in how you handle missing data? Diagnostics from your missing data strategy. And any sensitivity analyses that you do. [ONSCREEN CONTENT SLIDE 27] Demonstration in R [Alex Roehrkasse] We'll start now to transition to a demonstration in R. [ONSCREEN CONTENT SLIDE 28] Additional resources [Alex Roehrkasse] But before we do, I want to leave you with a couple additional resources to deepen and extend your learning on this topic. Both of these resources are resources I use myself and have relied on extensively. Both of them are open source and freely available on the internet, and both of them have extended examples and applications using R. [ONSCREEN CONTENT SLIDE 29] Questions? User support: NDACANsupport@cornell.edu Garrett baker: garrett.baker@duke.edu Alyssa Lindsey: Alyssa.lindsey@ucsf.edu [Alex Roehrkasse] With that, we'll start to transition to a demonstration in R and RStudio. Bear with me for a moment as we do that. Okay, hopefully by now this interface should be somewhat familiar. We're using R implemented in RStudio. Up here in the left, we'll be working through an R script specific to today's presentation. Today, we'll be doing a lot of visualization, which will be the theme of next week's presentation. So we'll be skipping over some of the code that actually generates these visualizations, but I want to draw your attention to this lower quadrant, where we'll be looking often to understand our approach to missing data. [ONSCREEN CONTENT] NOTES This program file demonstrates strategies discussed in session 4 of the 2026 NDACAN Summer Training Series "Handling missing data." For questions, contact the presenter Alex Roehrkasse (aroehrkasse@butler.edu). Note that because of the process used to anonymize data, all unique observations include partially fabricated data that prevent the identification of respondents. As a result, all descriptive and model-based results are fabricated. Results from this and all NDACAN presentations are for training purposes only and should never be understood or cited as analysis of NDACAN data. [Alex Roehrkasse] Today's presentation is or the script today is specific to today's presentation. A reminder that all of the data we'll be using today are fabricated. They have similar properties to real data, but we've fabricated them in order to protect the anonymity, the identity of actual children in archived datasets. So any analyses we do do not represent analyses of archived data. I'm gonna skip over our setup today. Please refer to previous presentations for information about how to set up your working environment. [ONSCREEN CONTENT] > ## SETTING UP THE ENVIRONMENT ## > > # Let's clear the environment. > rm(list=ls()) > > # Pacman installs packages if necessary, otherwise loading them. > if (!requireNamespace("pacman", quietly = TRUE)){ + install.packages("pacman") + } > pacman::p_load(data.table, tidyverse, + ggstance, + mice, ggmice) # new missing data packages > > # Let's define some filepaths > # (note the organization of project and data folders). > project <- 'C:/Users/aroehrkasse/Box/Presentations/-NDACAN/2026_summer_series/' > data <- 'C:/Users/aroehrkasse/Box/NDACAN/2026_summer_series/' > > # And set one as the working directory. > setwd(project) > > # Always set a seed to allow for reproduction of random processes. > # Especially important with mice package. > set.seed(1013) > # Let's read in our cleaned > # anonymized versions of the > # NCANDS Child Files for 2020-2024 (see session 2). > nc <- read_rds(paste0(data,'ncands_clean.rds')) > > # Let's also read in our cleaned, linked data: > # children 0-3 entering foster care in 2023 (AFCARS) > # linked to maltreatment histories (NCANDS) (see session 3). > dlink <- read_rds(paste0(data,'linked_data.rds')) [Alex Roehrkasse] We'll focus today's presentation on investigating and handling missing data. Okay, we'll go ahead and set up our environment. And we're gonna read in two datasets for today's presentations. Both of these datasets we've created in prior weeks' sessions The first is a stacked. NCANDS child file for 2020 to 2024. The second is a linked dataset in which we link path AFCARS namely, admissions into foster care in fiscal year 2023 to maltreatment histories measured using NCANDS data. When we read these objects in, we see we have two datasets here in our environment. A final reminder that for the purposes of presentation today, I've sampled only New England states and a small number of key variables. Because many missing data strategies are computationally intensive, we'll be further sampling our data just to keep things moving in today's presentation. Okay, how do we investigate missing data? How do we understand the extent of missing data and patterns of missing data in our datasets? Let's first tackle the case of record-level missingness. Honestly, this can often take quite a lot of elbow grease. [ONSCREEN CONTENT] > # Identifying record-level missingness can take elbow grease. > # As a starting point, it's usually helpful to plot > # record counts by reporting unit (usually state) > # and reporting period (usually submission year). > nc |> + count(staterr, subyr) |> + ggplot(aes(x = subyr, y = n)) + + geom_point() + + geom_line() + + facet_wrap(~staterr, scales = 'free') + + theme(axis.text.x = element_text(angle = 90)) [ONSCREEN CONTENT R CODE IMAGE 1] Six plots of records counts by state for CT, MA, ME, NH, RI, and VT from 2020-2024. Exploring record-level missingness. First, inspect plots of records counts by reporting unit (usually the state) and reporting period for trend breaks which indicate record-level missingness. [Alex Roehrkasse] As a starting point, it's usually helpful to plot Counts of records by reporting unit, usually the state, and reporting period, usually the submission year. This chunk of code will do this. Generating a line plot. that shows trends, For the count of record for each submission year for each state. As you can see, things go up, things go down, but none of these counts are so high or so low that we would expect some record-level missingness. If we were to do this for a larger number of states over a larger number of years, I can almost guarantee you that we would see trend breaks indicative of record-level missingness. [ONSCREEN CONTENT] > # Recall that using Child Files as a proxy for > # fiscal years will yield missing records toward the > # end of the FY due to delayed reporting. > nc |> + filter(rptdt %between% + c('2019-10-01','2020-9-30')) |> + group_by(rptdt, subyr) %>% + summarise(n = n(), .groups = 'keep') |> + ungroup() |> + ggplot(aes(x = rptdt, y = n, fill = fct_rev(as.factor(subyr)))) + + geom_col(position = 'stack') + + labs(fill = 'Submission year', x = 'Date', y = 'Sampled records') + + scale_x_date(date_breaks = "1 month", date_labels = "%b %Y") + + theme(axis.text.x = element_text(angle = 90, + hjust = .5, vjust = .5)) [ONSCREEN CONTENT R CODE IMAGE 2] A bar chart in showing the number of records observed in a sample of NCANDS data every half-month in fiscal year 2020. In most half-months, most records come from the 2020 NCANDS Child File, but toward the end of the year, a meaningful proportion come from the 2021 NCANDS Child File. [Alex Roehrkasse] Recall also that using child files as a proxy for fiscal year Can yield missing records toward the end of the fiscal year due to delayed reporting. This chunk of code pulls out records for the 2020 fiscal year, groups them by submission year, and generates a bar chart similar to the one I showed you in our presentation. Where here, all of these red areas are maltreatment reports occurring in the 2020 fiscal year, but not recorded until the 2021 child file. Without accounting for this delayed reporting, we can have record-level missingness in our data. [ONSCREEN CONTENT] > # Item-level missingness is a little more straightforward. > # Let's quickly search for missing values > # in our NCANDS Child Files: > # What percentage of each variable has missing values? > nc |> + summarize_all(~sum(is.na(.))) |> + mutate_all(~round(./nrow(nc)*100,2)) |> + t() [,1] subyr 0.00 staterr 0.00 stfcid 0.00 afcarsid 0.00 rptid 0.00 chid 0.00 rptdt 0.00 rptdisp 0.11 rpdispdt 0.00 chprior 0.00 fcmoney 88.69 per1rel 73.55 per1rel_new 73.85 [Alex Roehrkasse] Let's move on to item-level missingness. This is honestly a little more straightforward. As a first pass, we can take our child file, ase the Summarize All command to identify all those variables that have missing values, and to count up the percentage of variables, or percentage of observations, with missing values for each variable. When we do this, we see that many variables, most variables indeed, have no missing values. Certain variables, like report disposition, have 0.1% of records with missing values. But certain variables, like this financial distress variable, and also the perpetrator relationship variable, have a very high percentage of missing values. We're going to need to deal with this in a thoughtful way. A reminder that searching for any values, as we just did, only works if you've already properly cleaned your data, and I would refer you back to Session 2 for guidance about how to do that. [ONSCREEN CONTENT] > # Recall that searching for NA values only works > # if you've already properly cleaned your data (see session 2). > # Don't trust the codebook blindly: > # verify all variable encodings yourself. > nc |> + count(per1rel) # original: missing data incorrectly encoded per1rel n 1: 1 158461 2: 2 5924 3: 3 248 4: 4 370 5: 5 923 6: 6 830 7: 7 11938 8: 8 1407 9: 9 714 10: 10 484 11: 33 18 12: 88 6428 13: 99 2202 14: NA 528091 [Alex Roehrkasse] Don't trust the codebook blindly. Verify all variable encodings yourself. As an illustration, let's look at how the original perpetrator relationship variable is encoded. As we would expect, it has these different numerical values. But toward the bottom, we see that about 2,000 records have a code 99, which is how the codebook says missing data should be encoded. But another 528,000 records have no value at all. This is not what the codebook would indicate, although this is how R needs to understand the missing data. What we really need to do is recode these 2,000 observations as missing in a way that R will understand it. [ONSCREEN CONTENT] > nc |> + count(per1rel_new) # clean encoding per1rel_new n 1: Foster parent 636 2: Friend/neighbor 484 3: Legal guardian 1407 4: Other 6428 5: Other relative, non-foster 5924 6: Parent 158461 7: Professional 2467 8: Unmarried partner of parent 11938 9: 530293 [Alex Roehrkasse] If we look at our cleaned version of this variable that we created in session 2, we see that we've already solved this problem by explicitly recoding all missing values in a way that R will understand. [ONSCREEN CONTENT] ## EXPLORING THE MISSING-DATA MECHANISM ## # As discussed, the missing-data mechanism matters. # While it's not usually possible to observe it directly # from the data alone, we can make inferences about it # using missing-data patterns. # For example, plotting variable distributions by # state and year can help identify how much missingness # is a function of the state or state-year of report. > nc |> + count(staterr, subyr, fcmoney) |> + group_by(staterr, subyr) |> + mutate(p = n/sum(n)) |> + ggplot(aes(x = p, y = staterr, fill = fcmoney)) + + geom_histogram(stat = 'identity') + + facet_wrap(~ subyr) + + theme(axis.text.x = element_text(angle = 90, + vjust = .5)) Warning message: In geom_histogram(stat = "identity") : Ignoring unknown parameters: `binwidth` and `bins` [Alex Roehrkasse] What about exploring the missing data mechanism? As I said, this mechanism matters, but it's not usually possible to observe it directly. We can, though, make inferences about it using patterns in our missing data. One helpful way to do that is to plot variable distributions By state and by year. Which can help us identify how much missingness is a function of the state or state year from which the data come. This chunk of code will do just that. Grouping things by state, year, and value of a variable of interest, and then plotting the distributions. [ONSCREEN CONTENT R CODE IMAGE 3] Plot of the fcmoney financial distress variable (with values yes, no, NA) for fiscal years 2020 through 2024 for six states. [Alex Roehrkasse] What we see here is that for each year, each submission year, and for each state, we have a certain proportional distribution of the values of the fcmoney variable, the financial distress variable. Obviously, for certain states, like Rhode Island, New Hampshire, Maine, Connecticut, the variable is never observed. It's completely missing from those states strongly indicative that those states simply don't record that information. In Massachusetts, we see that the years 2020, 2021, and 2024 have very similar distributions, but in 2022 and 2023, the scenario is much different. This gives us reason to suspect that the missingness for this variable results from changes in state-level data collection and reporting protocols. It also suggests that the values for this variable are mostly missing at random, rather than missing completely at random, or missing not at random. We would want to be transparent about this in any analysis of missing data that we did. [ONSCREEN CONTENT] > # Note that in the case of linked data, > # *item*-level missingnesscan arise from > # *record*-level missingness. > # Examining our linked dataset, we see that a > # (very small) number of observations have missing values > # for maltreatment history because these children had > # missing records of maltreatment. > dlink |> + summarize_all(~sum(is.na(.))) |> + mutate_all(~round(./nrow(nc)*100,2)) |> + t() [,1] fy 0.00 st 0.00 recnumbr 0.00 dob 0.00 totalrem 0.00 rem1dt 0.00 latremdt 0.00 ageatlatrem 0.00 dodfcdt 0.32 entered 0.00 exited 0.00 inatend 0.00 inatstart 0.00 stfcid 0.00 version 0.37 nsub 0.03 nrep 0.03 maxdt 0.03 [Alex Roehrkasse] Lastly, I want to note that in the case of linked data, item-level missingness can arise from record-level missingness. If we look at our linked dataset, we see, for example, that a very small, but nevertheless, you know, potentially interesting proportion of observations have missing values for our maltreatment history variable. This is because we failed to link these records. And we failed to link these records because there was record-level missingness in NCANDS. In the case of linked data, record-level missingness gives rise to item-level missingness. [ONSCREEN CONTENT] > # Let's explore options for handling missing data > # using maltreatment reports for FY2020-2024. > # Let's also create an indicator variable measuring > # whether a report was substantiated or indicated. > ncs <- nc |> + filter(rptdt %between% + c('2019-10-01','2023-9-30')) |> + mutate(subind = case_when( + rptdisp %in% c('Substantiated', + 'Indicated/reason to suspect') ~ 1, + is.na(rptdisp) ~ NA, + T ~ 0) + ) [Alex Roehrkasse] Okay, this brings us to handling missing data. What we actually do about the challenge of missing data. We'll explore some options for handling missing data using maltreatment reports for the fiscal years 2020 through 2024. To do this, we'll also create an indicator variable, which just measures whether or not a report was substantiated or indicated, or not. So we'll go ahead and create this new version of our dataset. And we'll talk through a few different approaches to handling missing data. [ONSCREEN CONTENT] # COMPLETE-CASE ANALYSIS # # Let's estimate a very basic logistic regression model, # and summarize it. Counting the observations used in the model, # note that it differs from the length of the full dataset. # This is because most models can't be estimated directly on # observations with missing values of modeled variables. # In other words, the default is to conduct a # complete-case analysis, or to listwise-delete # observations with missing values. > m_cc <- glm(subind ~ chprior + fcmoney, + data = ncs, + family = 'binomial') [Alex Roehrkasse] First being complete case analysis, otherwise known as listwise deletion. Let's say our estimand is, we're trying to estimate model parameters in a logistic regression model. We want to know what, explanatory variables predict substantiation or indication. We can do this using the GLM function, where we estimate a binomial or logistic regression model using our dataset where we're predicting substantiation or indication as a function of whether or not the child had a prior report, and whether or not they experienced financial distress. Let's go ahead and estimate that model. Summarize it. [ONSCREEN CONTENT] > summary(m_cc) Call: glm(formula = subind ~ chprior + fcmoney, family = "binomial", data = ncs) Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) 0.10250 0.03513 2.918 0.00353 ** chpriorNo 0.03731 0.01857 2.009 0.04451 * fcmoneyNo -0.59147 0.03475 -17.020 < 2e-16 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 75424 on 56193 degrees of freedom Residual deviance: 75133 on 56191 degrees of freedom (522063 observations deleted due to missingness) AIC: 75139 Number of Fisher Scoring iterations: 4 > nobs(m_cc) [1] 56194 > nrow(ncs) [1] 578257 [Alex Roehrkasse] And we see something that looks somewhat like a regression table, if you've ever seen one reported in, say, an academic journal article. But notice something that has happened. When we look at the number of observations used in the estimation of this model, it's considerably smaller, about one tenth of the number of observations in our dataset. This is because, by default, R, when estimating a statistical model, list-wise deletes any observations with missing values for variables that are part of the model. Recall that this is only defensible if our data are missing completely at random and as we've already shown, there's good reason to think they're not. What else can we do? [ONSCREEN CONTENT] # HOT DECK IMPUTATION # # Hot deck imputations fills in missing values # using observed values from other units with # similar observed values. Usually it's not a very good way # to impute missing data. But the NCANDS Child Files # (and other NDACAN data) have a special property that makes it # suitable: repeated observations of the same units, in this case, # multiple reports for many of the same children. Perhaps fcmoney # is missing for a child on one report but not on another. # So let's sort by child ID and then report date. [Alex Roehrkasse] As I mentioned, hot DECA imputation is somewhat outdated, but there's reasons to take it seriously in the case of archive administrative data. This is because our data have a special property that makes it more powerful. Namely, repeated observations of the same unit. In this case, multiple reports for many of the same children. There may be some child attributes that we have reason to think are consistent over time. If they're measured at one point in time, but not at another, maybe we can borrow information about this same child from another observation about them and fill it in. Let's try this. [ONSCREEN CONTENT] > ncs |> + arrange(stfcid, rptdt) subyr staterr stfcid afcarsid rptid 1: 2022 CT CT000410019186 410019186 100005B45C09 2: 2021 CT CT000410056939 410056939 1000057AF159 3: 2020 CT CT000410059634 410059634 1000053A36B1 4: 2020 CT CT000410072065 410072065 1000053A3338 5: 2021 CT CT000410513558 410513558 1000057AC1F6 --- 578253: 2020 VT VT999822899999 999822899999 1000053D719F 578254: 2021 VT VT999822899999 999822899999 10000568BCDB 578255: 2020 VT VT999912899999 999912899999 1000053D7766 578256: 2022 VT VT999912899999 999912899999 100005AF3D3D 578257: 2020 VT VT999922899999 999922899999 1000053D81A7 chid rptdt rptdisp rpdispdt chprior 1: 100001862356 2022-04-08 Substantiated 2022-04-08 Yes 2: 100001816BC8 2021-01-23 Substantiated 2021-02-14 Yes 3: 10000184E831 2020-01-23 Unsubstantiated 2020-03-04 Yes 4: 10000539F44A 2020-02-08 Unsubstantiated 2020-03-06 No 5: 1000057A811C 2020-12-23 Unsubstantiated 2021-01-15 No --- 578253: 1000053D6607 2019-12-23 Unsubstantiated 2020-01-27 No 578254: 1000053D6607 2021-09-08 Unsubstantiated 2021-09-17 No 578255: 1000053D6609 2020-03-08 Substantiated 2020-03-31 No 578256: 1000053D6609 2022-05-23 Unsubstantiated 2022-06-02 Yes 578257: 1000053D660A 2019-12-08 Unsubstantiated 2020-03-04 No fcmoney per1rel per1rel_new subind 1: 1 Parent 1 2: NA 1 3: NA 0 4: NA 0 5: NA 0 --- 578253: No NA 0 578254: No NA 0 578255: No 1 Parent 1 578256: No NA 0 578257: No NA 0 > # And count rows representing the same child and for which > # fcmoney is missing in one row but not another. > ncs |> + filter(is.na(fcmoney) & !is.na(lag(fcmoney)) & + stfcid == lag(stfcid) & + chid != 'XXXXXXXXXXXX') subyr staterr stfcid afcarsid rptid chid 1: 2022 VT VT582102899999 582102899999 100005AF3B72 10000568A272 rptdt rptdisp rpdispdt chprior fcmoney per1rel 1: 2022-01-23 Unsubstantiated 2022-03-01 Yes NA per1rel_new subind 1: 0 [Alex Roehrkasse] First, we take our, NCANDS child files data object, and we arrange them, or order them. First, by the child's unique ID, and then the report date. Then, we want to count up all of the rows for which it's the case that the money variable is observed, or missing in one case, but observed in the other, ensuring that we're looking at the same child. Is this the case? Well, yes. Only for one child, who happened to live in Vermont in 2022. Not much, but not nothing. And I'm willing to bet that if you implemented this strategy with other variables on a larger sample, you'd find a non-trivial number of cases where you can carry over values for the same child observed multiple times. Now, the validity of this approach rests on the assumption that children's true financial distress is time invariant. That's probably false in this case, but for other variables, it might be a defensible solution. [ONSCREEN CONTENT] > # Bingo! Just one, but not nothing, and likely to be more common > # depending on the sample and variables analyzed. > # So let's carry over the non-missing values > # to the missing ones. Validity rests on the assumption that > # children's true financial distress is time-invariant. > # (This assumption is actually false, > # but might not be in other cases.) > ncs <- ncs |> + group_by(stfcid) |> + fill(fcmoney, .direction = 'downup') |> + ungroup() [Alex Roehrkasse] We can then carry over observations, both up and down, Within a particular child, partially handling at least a small number of missing values. Okay, this brings us to multiple imputation. As I said, the kind of bread and butter approach in modern missing data analysis. [ONSCREEN CONTENT] # MULTIPLE IMPUTATION # # Multiple imputation is a model-based strategy that uses # information about the relationships between observed values # to make guesses about the true value of missing values. # We make multiple guesses, or imputations, the variance of which # captures our uncertainty about the accuracy of our model. # For demonstration purposes, let's take a random 3% sample # or our data, keeping only those variables of interest. # Note that multiple imputation is computationally intensive. # Plan to have your machine impute your data while you # eat lunch or sleep. Consider the need for non-local computing. > ncsi <- ncs |> + select(subyr, staterr, rptdt, stfcid, + subind, + chprior, fcmoney) |> + mutate(chprior = factor(chprior, + levels = c('No', 'Yes'), + labels = c('No', 'Yes')), + fcmoney = factor(fcmoney, + levels = c('No', 'Yes'), + labels = c('No', 'Yes'))) |> + slice_sample(prop = .03) [Alex Roehrkasse] Multiple imputation is a model-based strategy that uses information about the relationships between observed variables to make guesses or inferences about the true values of missing variables. We make multiple guesses or imputations, the variance across which captures our uncertainty about the accuracy of our model. Now, multiple imputation is a complicated topic we're just going to be scratching the surface here, trying to walk through an implementation of this approach. I want to be very clear that if you were to actually implement this in your research project, there are a variety of other things you would want to think carefully about. We're happy to support you in that through consultation. Today is just some foundational skills. For demonstration purposes, because multiple imputation is computationally intensive, we'll just take a random 3% sample of our data, keeping only variables of interest. As I said, because it's computationally intensive, it's often helpful to do this first with a small sample of your data. When you do this with a full dataset, plan on needing to run your imputations while you eat lunch or overnight. Sometimes you may need to implement these kinds of strategies on a server or some sort of cloud computing resource. Okay, so let's first clean up our data a little bit more. We're just going to recode a couple factors to make them more interpretable further down. And now we'll examine the missingness in this sample. Not just the extent of missingness, but the pattern of missingness. [ONSCREEN CONTENT] > # We examine patterns of missingness in the sample, > # which is important for assessing the feasibility > # of multiple imputation. > ncsi |> + plot_pattern(rotate = T) + + theme(axis.text.x = element_text(vjust = 0.25), + legend.position = 'none') [ONSCREEN CONTENT R CODE IMAGE 4] Heatmap showing missing data patterns across seven columns: sublvr, stater, protd, strcid, chprior, subind, and fcmoney. The x-axis is “Number of missing entries per column” and the y-axis is “Pattern frequency.” Most cases have zero or few missing values, while the bottom-right cells show patterns with missing entries in subind and fcmoney. A note below says the total number of missing entries is 15,294. [Alex Roehrkasse] What do I mean by a missing data pattern? I mean, how often is it the case that certain variables have co-occurring missing values? We see here that we have essentially 3 different missing data patterns. The first pattern is where no data are missing. We completely observe all of the variables of interest. The second scenario, the most common scenario, is where we have one missing value for the fcmoney variable. So for 15,000 records, it's the case that we have one missing value. This is a scenario where we can use information from the observed values to make guesses about the missing value. The third missing data pattern is where we also have missing data for the substantiation indication variable. Understanding patterns of missingness in your data, not just the extent of missing data, is important to assess the feasibility of multiple imputation. [ONSCREEN CONTENT] > # And designate variables we *don't* want to use in the imputation. > pred <- quickpred(ncsi, + exclude = c("stfcid", "rptdt")) [Alex Roehrkasse] Now, something slightly technical, we'll designate some variables that we don't want to use in our imputation, namely specific child IDs and the specific dates of certain reports. And then we will go ahead and estimate an imputation model. You may notice that nowhere in here do we actually specify a formula, or specify a model. The mice command, which estimates the model, does this intuitively. There may be good reasons that you want to specify specific features of the imputation model, and mice allows for this. But briefly, what we'll do is tell it to use our dataset, our main dataset. Tell it to estimate 5 imputations, which, as a rule of thumb, is a good minimum number. And we'll go ahead and estimate this, this imputation model. [ONSCREEN CONTENT] > # Then we estimate an imputation model. > imp <- mice(ncsi, + predictorMatrix = pred, + m = 5, + maxit = 5, + print = F) Warning message: Number of logged events: 1 [Alex Roehrkasse] We'll see that it creates a new object over here in our environment. Namely, a multiple imputation data object. A MIDS object. We can check the basic features of our imputation using the summary command. [ONSCREEN CONTENT] > # We can check basic features of the imputation. > # Note that we probably wanted mice to use "logreg" for subind > # instead of "pmm." We can either recode the variable, > # or specify imputation methods using the "method" option of mice. > summary(imp) Class: mids Number of multiple imputations: 5 Imputation methods: subyr staterr rptdt stfcid subind chprior fcmoney "" "" "" "" "pmm" "" "logreg" PredictorMatrix: subyr staterr rptdt stfcid subind chprior fcmoney subyr 0 0 0 0 0 0 0 staterr 0 0 0 0 0 0 0 rptdt 0 0 0 0 0 0 0 stfcid 0 0 0 0 0 0 0 subind 0 0 0 0 0 1 0 chprior 0 0 0 0 0 0 0 Number of logged events: 1 it im dep meth out 1 0 0 constant staterr [Alex Roehrkasse] Especially helpful here is to see what kind of estimation mice chose to use to estimate each variable. Note that it intuitively chose the predictive mean-matching approach for this variable, which might not be advisable. This would be something we might re-specify using the method option. We can also very simply do some basic diagnostics. [ONSCREEN CONTENT] > # We can easily diagnose the convergence of our imputation model > # (see additional resources). > plot(imp) [ONSCREEN CONTENT R CODE IMAGE 5] Four small trace plots tracking imputation estimates over iterations. The panels show the mean and standard deviation for two variables, subind and fcmoney. Multiple colored lines fluctuate slightly across iterations, suggesting the estimates are converging. The x-axis is labeled “Iteration.” [Alex Roehrkasse] These are trace plots for our imputation models. Essentially seeing whether certain statistical assumptions are satisfied for valid imputation. As I said up a little earlier, there's much more to learn about mice. For example, based on our missing data patterns, we may want to predict missing values not just using state and year, but specifically a state-year interaction. At very least, I encourage you to explore the help documentation for this package. There are a number of vignettes for mice that very helpfully demonstrate use cases that may be generalizable to our archive data. [ONSCREEN CONTENT] > # Let's now use our imputed data to estimate the same, > # basic logistic regression model that we estimated > # on our complete-case data. > m_mice <- with(imp, + glm(subind ~ chprior + fcmoney, + family = 'binomial')) > # And visually compare the results for the two models. > # First, re-estimate our complete-count model on the same > # random 3% sample. > m_cc2 <- glm(subind ~ chprior + fcmoney, + data = ncsi, + family = 'binomial') [Alex Roehrkasse] For now, though, let's go ahead and use our imputed dataset to estimate this same Logistic regression model. Generating some model results as we do. Now, let's compare the results for our mice model and our complete count model. We need to re-estimate our complete count model using the same sample we used for mice. I will skip over the following code because it's not of substantive interest to today's demonstration. Essentially, I'm just organizing the estimates in a specific way. And then lastly, I will combine the estimates. And generate a plot, That compares them. [ONSCREEN CONTENT] > # Then organize the estimates. > cc_summary <- summary(m_cc2) |> + coef() |> + as.data.frame() |> + rownames_to_column('term') |> + mutate(model = 'CC') |> + rename(est = Estimate) |> + select(term, est, model) > cc_ci <- confint(m_cc2) |> + as.data.frame() |> + rownames_to_column('term') |> + rename(lower = `2.5 %`, upper = `97.5 %`) Waiting for profiling to be done... > cc_combined <- left_join(cc_summary, cc_ci, by = 'term') > # Organize the mice estimates. > mice_combined <- pool(m_mice) |> + summary(conf.int = TRUE) |> + as.data.frame() |> + mutate(model = 'MICE') |> + rename(est = estimate, lower = `2.5 %`, upper = `97.5 %`) |> + select(term, est, lower, upper, model) > # And finally, combine and visually compare the estimates. > bind_rows(mice_combined, cc_combined) |> + mutate(term = factor(term, + levels = c('(Intercept)', + 'chpriorYes', + 'fcmoneyYes')), + est = exp(est), + lower = exp(lower), + upper = exp(upper)) |> + filter(term != '(Intercept)') |> + ggplot(aes(x = est, y = fct_rev(term), + xmin = lower, xmax = upper, + color = model, group = model)) + + geom_vline(xintercept = 1, linetype = 'dashed') + + geom_point(position = position_dodgev(height = -.5)) + + geom_errorbarh(height = .25, + position = position_dodgev(height = -.5)) + + labs(x = 'Odds ratio', y = 'Variable', color = 'Model') Warning messages: 1: position_dodgev requires non-overlapping y intervals 2: Using the `size` aesthetic with geom_path was deprecated in ggplot2 3.4.0. ℹ Please use the `linewidth` aesthetic instead. This warning is displayed once per session. Call lifecycle::last_lifecycle_warnings() to see where this warning was generated. [ONSCREEN CONTENT R CODE IMAGE 6] Forest plot comparing odds ratios for two variables, chpriorYes and fcmoneyYes, under two models: CC and MICE. A dashed vertical reference line marks an odds ratio of 1.0. The CC estimate for chpriorYes is slightly below 1, while the MICE estimate is above 1.5. For fcmoneyYes, the CC estimate is around 1.5 with a wide confidence interval, and the MICE estimate is close to 1.0 with a moderate confidence interval. [Alex Roehrkasse] What do we see? Okay. Here, our estimand is an odds ratio. By what extent do each of these predictor variables increase or decrease the odds that a maltreatment report is substantiated or indicated. Our model estimates for the complete count model are in red. Our estimates for the mice model are in blue. The circle is the point estimate. And the tie fighter lines are 95% confidence intervals. The dotted line at 1 indicates no effect. When we look at the results for the complete count model, we see that there's not a statistically significant effect of prior report on substantiation. Whereas, we do find a statistically significant and fairly large effect of financial distress on substantiation. How do things change when we look at our results based on our multiply imputed data? Well, almost the exact opposite. Now we see a large and fairly precisely estimated effect for prior maltreatment report on substantiation risk. Whereas the previously observed effect for financial distress goes away entirely. Obviously, these differences in inference are meaningful. A reminder that these data are completely fabricated, so please don't take from this example any substantive claims about actual maltreatment in the United States. It's just to illustrate the main point, namely that improper handling of missing data can meaningfully bias our analysis and lead us to false inferences about our research question. This brings me to the end of my presentation, although, just one final acknowledgement that we've focused on foundational skills today, really just skimming the surface. I'll bring up our slide deck again and remind you that we are always here to help you, in your research pursuits, my hope is that, and I'm seeing in the Q&A, there's a number of questions we can discuss with our remaining time, but of course, if a question doesn't arise in the next 10 minutes or so, you think of one an hour from now, a day from now, a year from now, please don't hesitate to reach out, either to the user support line or to me personally. We're here to help with this and other aspects of data use. Thanks again, and I look forward to your questions. [ONSCREEN CONTENT SLIDE 29] Questions? User support: NDACANsupport@cornell.edu Garrett baker: garrett.baker@duke.edu Alyssa Lindsey: Alyssa.lindsey@ucsf.edu [Alyssa Lindsey] Thank you so much, Alex. I'm gonna go ahead and put in the chat for everyone the frequently asked questions tab of our website as well, as you're going through and have any questions. [ONSCREEN CONTENT] COMMENTS IN THE ZOOM CHAT Email the NDACAN archive at NDACAN@cornell.edu with general questions, and email NDACANsupport@cornell.edu with technical support questions. The NDACAN archive website https://www.ndacan.acf.hhs.gov contains the following: The R code and output of this presentation. Information and instructions for joining the Child-Maltreatment- Research-L (CMRL) list serve. The FAQ Frequently Asked Questions about using the NDACAN archive data including AFCARS, NCANDS, and NYTD. Instructions for ordering restricted datasets [Alyssa Lindsey] But let's go ahead and jump into the questions in the Q&A. The first one that came in, do you have some examples of internal documentation information about record-level missingness that can be provided? For example, can you provide documentation related to data suppression due to tribal agencies in the NCANDS files? [Alex Roehrkasse] This is a great question. Thank you, Lauren. Yeah, for the most part, what NDACAN can provide is, basically versions of the table I showed you in my slides, where we can make it quick and easy for you to see which states are reporting, in which years. I would encourage you to verify this independently yourself. But basically some spreadsheets that would help you, focus your attention on, non-reporting from different states. The tribal agency question is a trickier one. It's one that's come up in my own research in the last year. Title IV agencies, Title IV tribal agencies are required to report to the Children's Bureau. There aren't many of these agencies, but no, the archive itself does not receive that information. That information is retained with the Children's Bureau. And so the Children's Bureau, in that specific case, would be your point of contact. In my specific research question, where we were trying to understand the degree by which we may be undercounting child welfare system contact for American Indian Children, we relied on a newer resource that the Children's Bureau has created, the AFCARS Dashboard, which does release some aggregate information about tribal agencies, for those agencies that are required to report. We can triangulate from resources like that to make inferences about missing data under certain assumptions. I'm happy to talk about that question further, but for the most part, any questions specifically about data suppression from tribal agencies, I'd refer you to the Children's Bureau directly. [Alyssa Lindsey] Great, thank you, Alex, and thanks for this question. The next question says, given the state's example for the pattern of item-level missingness, would national or state-to-state comparisons with the data mean that it might have the caveat that states reporting much more complete data are likely to show increased trends over states that have missing data? [Alex Roehrkasse] Yeah, I understand the question to be does the degree of non-reporting affect inferences we would make about trends. The short answer is yes, but it's not, prima facie obvious, like, what that directional effect would be. And so it's one of the reasons why we implement some of these more sophisticated strategies we've talked about. And it, again, goes back to what your estimand is. If you're trying to calculate, for example, a population rate of child welfare system contact, then missing data has a very strong effect on your estimate, because every missing record is downward bias in your estimated rate. If you're trying to estimate, like, a model parameter, then there's significant potential for bias, but it's not as straightforwardly obvious that missingness will bias your results. So, I would encourage you to think about, what is your estimand? In this case, it seems to be trends over time. I'm guessing trends in, say, counts or incidence rates. In this case, yes, record-level missingness will definitely influence your estimates. Item-level missingness would as well, as long as the missing item, you know, pertains directly to your estimated rate. But the direction, how that would affect trends would depend on trends in the missing data mechanism. Would depend on how missing this is correlated with the underlying true values of the variable you're trying to track. So it's a complicated question and a complicated answer. I'll try to sum it up by saying, yes, item- and record-level missingness would affect your estimates. It's not obvious up front how it would, and you would need to investigate that empirically. [Alyssa Lindsey] Great, thank you, and thanks for this question. Related to this question, they ask, additional question related to above, would the AFCARS Dashboard already account for this? So if you were to use that data to compare states, is that possible, or is it always best to analyze states in isolation? [Alex Roehrkasse] Yeah, this is a great question. My recollection is that the Children's Bureau and the Archive are both quite careful whenever we're reporting estimates that are based on data coming from less than the entire U.S. population, we acknowledge as much. And so, sometimes, data users or readers of that literature aren't always noting those caveats, but it's important to note those caveats. It really depends on what your analytic goals are. If the data published, say, in the Children's Bureau, you know, Child Maltreatment reports, or AFCARS reports, or through the AFCARS Dashboard, those kinds of aggregate quantities are adequate to your research goals. then that can often be a very useful source of data, as long as you're noting carefully those cases where those national estimates are based on fewer than the whole country. Maybe some states are missing from those estimands. If you are comfortable using the data we distribute ourselves, it offers the opportunity to more directly verify and more fully understand the extent to which we're actually measuring the things you're interested in completely. I'm happy to answer the question more fully, over email. It may depend specifically on what you're trying to analyze. [Alyssa Lindsey] Great, thank you so much. And yes, I put our emails in the chat. Are there any other questions? Feel free to put them in the Q&A box. Again, that's in the lower right-hand corner of the screen. Okay, not seeing any come in. Maybe, Alex, we can transition to what next week looks like. I'll pass it to you. [ONSCREEN CONTENT SLIDE 30] Next week… Date: July 29th Topic: Data presentation and visualization Instructor: Alex Roehrkasse [Alex Roehrkasse] Great. Thanks, Alyssa. So, next week is our last presentation in this year's summer training series. I'm particularly excited about it. We'll be talking about data presentation and visualization. Focusing on how, once you've done an analysis, you can most effectively present the results of that analysis to communicate your findings to different audiences. We'll be tying together a lot of different stuff we've been talking about all summer, but again, each of these presentations stands alone, so feel free to join us, no matter how many other sessions you've attended. I want to thank everyone again for your attention, and your time. Have a great rest of your day. [Alyssa Lindsey] Thank you, everyone. Goodbye. [VOICEOVER] The National Data Archive on Child Abuse and Neglect is a joint project of Duke University, Cornell University, University of California, San Francisco, and Mathematica. Funding for NDACAN is provided by the Children's Bureau, an office of the Administration for Children and Families. [Music]