Child-Maltreatment-Research-L (CMRL) List Serve

Message ID: 9017

Date: 2011-12-03

Author:Wells, Susan J

Subject:RE: Measuring racism

Please forgive me if you have already mentioned this, but see also the work of Samuel L. Myers, Jr. for analyses conducted along these lines in defining racism and measuring bias in employment, housing, criminal justice as well as child welfare. Susan J. Wells -----Original Message----- From: bounce-38292065-12990613@list.cornell.edu [mailto:bounce-38292065-12990613@list.cornell.edu] On Behalf Of Chaffin, Mark J. (HSC) Sent: November 20, 2011 4:20 PM To: Child Maltreatment Researchers Subject: RE: Measuring racism Jane Marie and list members, I want to revisit Jane Marie's original question, namely how do we MEASURE (emphasis added) racism in child welfare systems. From a measurement perspective, bias is less often directly measured and more often inferred or estimated from something analogous to a quasi-experiment. If you look across studies, the main way of approaching this is by comparing case decision making across cases. The ideal and conclusive comparison would be this one: Correct decision (y/n) for Case A (observed minority race condition) vs. Correct decision (y/n) for Case A (non-minority race counterfactual condition). Notice that both are Case A. A n/y contrast between the observed and counterfactual conditions supports an inference of racial bias--that is, the wrong decision is made under the minority condition, but the right one is made under the non-minority counterfactual. A y/y or n/n contrast would not support bias; although the n/n contrast would be something even more concerning than bias--systematic failure for everyone. A y/n contrast would suggest reverse bias. The estimation challenge, of course, is twofold--one cannot observe outcomes under both conditions (i.e. one case cannot be both observed and counterfactual, both minority and non-minority, at the same time), and one must have some way of knowing what the correct decision should have been for the case. There are two quasi-experimental methods issues here, neither of which IMO has been handled with much rigor in the majority of the disparity research, for some understandable reasons. The first is that not nearly enough data has been available on the cases to create reasonable counterfactuals. Studies may use only a few crude covariates like neighborhood or single parenting or poverty status that don't really give us a clear picture of the individual case risks, problems and resources. For example, studies may make the tenuous assumption that individuals in two given neighborhoods with similar levels of poverty have similarly distributed sets of problems, risks and resources and therefore should have similar patterns of dispositions. Does anyone who has worked in neighborhoods actually believe this? The related issue is that studies to my knowledge have not considered what the correct decision should have been in a particular case. They simply look for disparity on some action, such as foster care placement, irrespective of what "should" have happened under accepted best practices. This leaves open the debate we commonly hear about whether higher African American foster placement means that child welfare is unfair to African American parents, or whether child welfare is appropriately protecting African American children. What we want to know is whether those placements were the correct decision vs. whether those children would have been better served within their families. Until we get better case-by-case data to judge what best practice should have dictated for a particular case profile, the question is difficult to answer. There is a statistical issue concerning the use of regression covariance methods in these types of studies. Covariance does not estimate counterfactuals or create apples-to-apples comparisons--it "adjusts" estimates for the effects of a potential confounder under a set of assumptions which may not be realistic for our question here, and is often limited to estimating effects only at the mean of the covariate. Its not always a strong method. As another poster noted, the impact of poverty on actual maltreatment behavior is not equivalent across ethnic groups, and so introducing poverty as a regression model covariate may potentially exaggerate bias estimates. What we want are homogeneous strata or even matched pairs with respect to the clinical and case type variables that we believe are important for making "correct" decisions--safety, options and resources, including group-specific consideration for some risk factors such as poverty, and then make comparisons within those homogeneous strata defined by best practices. For example, we want strata with similar placement need attributable to parental substance use or depression, availability of extended family or social network supports, seriousness of parent-to-child violence, parental capacity to care for the child, family violence, etc. Creating homogeneous strata to examine the issue of bias is complicated and does involve some thorny decisions. For example, if no good family preservation services exist in a minority community, do you balance strata on that variable, treat that variable as one element of structural bias and therefore part of the independent variable, or use it as a covariate? If poverty is a big risk factor for serious parent-to-child violence in one group, but not the other, do you weight poverty the same or differently across groups? Lastly, there may be a base-rate complexity to this question. For example, if true maltreatment base rates are high in group A, and lower in group B, then a really sharp worker actually should evaluate objective risk indicators differently across the two groups (i.e. the sensitivity and specificity of the risk indicators will not be equal between groups). This might need to be factored into determining what the "correct" decision in a case should have been, because it may or may not be equivalent across groups. For example, there is a higher prevalence of sex abuse perpetration among men than among women. Therefore, we might evaluate exactly the same allegation against a mother as less probable than we might if it was against a father. One could (and many of the father advocates in the world would) call this bias or unfairness. Others might just call it objective Bayesian reasoning. To my knowledge, this issue has never been incorporated into bias measurement or disparity research, including in the vignette studies. All of this means that measuring racial bias in child welfare decision making is a complex issue--in some ways more analogous to messy quasi-experiments than to psychometrics or sociometrics. The reality of racial bias, historically or currently, is not what is in question here, and by pointing out these estimation issues my intent is not to encourage either racism denial or racism exaggeration. Rather, the issue is the accuracy and precision of our estimates, and I think we have a ways to go to measure this precisely. Mark Chaffin

Please forgive me if you have already mentioned this, but see also the work of Samuel L. Myers, Jr. for analyses conducted along these lines in defining racism and measuring bias in employment, housing, criminal justice as well as child welfare. Susan J. Wells -----Original Message----- From: bounce-38292065-12990613list.cornell.edu [mailto:bounce-38292065-12990613list.cornell.edu] On Behalf Of Chaffin, Mark J. (HSC) Sent: November 20, 2011 4:20 PM To: Child Maltreatment Researchers Subject: RE: Measuring racism Jane Marie and list members, I want to revisit Jane Marie's original question, namely how do we MEASURE (emphasis added) racism in child welfare systems. From a measurement perspective, bias is less often directly measured and more often inferred or estimated from something analogous to a quasi-experiment. If you look across studies, the main way of approaching this is by comparing case decision making across cases. The ideal and conclusive comparison would be this one: Correct decision (y/n) for Case A (observed minority race condition) vs. Correct decision (y/n) for Case A (non-minority race counterfactual condition). Notice that both are Case A. A n/y contrast between the observed and counterfactual conditions supports an inference of racial bias--that is, the wrong decision is made under the minority condition, but the right one is made under the non-minority counterfactual. A y/y or n/n contrast would not support bias; although the n/n contrast would be something even more concerning than bias--systematic failure for everyone. A y/n contrast would suggest reverse bias. The estimation challenge, of course, is twofold--one cannot observe outcomes under both conditions (i.e. one case cannot be both observed and counterfactual, both minority and non-minority, at the same time), and one must have some way of knowing what the correct decision should have been for the case. There are two quasi-experimental methods issues here, neither of which IMO has been handled with much rigor in the majority of the disparity research, for some understandable reasons. The first is that not nearly enough data has been available on the cases to create reasonable counterfactuals. Studies may use only a few crude covariates like neighborhood or single parenting or poverty status that don't really give us a clear picture of the individual case risks, problems and resources. For example, studies may make the tenuous assumption that individuals in two given neighborhoods with similar levels of poverty have similarly distributed sets of problems, risks and resources and therefore should have similar patterns of dispositions. Does anyone who has worked in neighborhoods actually believe this? The related issue is that studies to my knowledge have not considered what the correct decision should have been in a particular case. They simply look for disparity on some action, such as foster care placement, irrespective of what "should" have happened under accepted best practices. This leaves open the debate we commonly hear about whether higher African American foster placement means that child welfare is unfair to African American parents, or whether child welfare is appropriately protecting African American children. What we want to know is whether those placements were the correct decision vs. whether those children would have been better served within their families. Until we get better case-by-case data to judge what best practice should have dictated for a particular case profile, the question is difficult to answer. There is a statistical issue concerning the use of regression covariance methods in these types of studies. Covariance does not estimate counterfactuals or create apples-to-apples comparisons--it "adjusts" estimates for the effects of a potential confounder under a set of assumptions which may not be realistic for our question here, and is often limited to estimating effects only at the mean of the covariate. Its not always a strong method. As another poster noted, the impact of poverty on actual maltreatment behavior is not equivalent across ethnic groups, and so introducing poverty as a regression model covariate may potentially exaggerate bias estimates. What we want are homogeneous strata or even matched pairs with respect to the clinical and case type variables that we believe are important for making "correct" decisions--safety, options and resources, including group-specific consideration for some risk factors such as poverty, and then make comparisons within those homogeneous strata defined by best practices. For example, we want strata with similar placement need attributable to parental substance use or depression, availability of extended family or social network supports, seriousness of parent-to-child violence, parental capacity to care for the child, family violence, etc. Creating homogeneous strata to examine the issue of bias is complicated and does involve some thorny decisions. For example, if no good family preservation services exist in a minority community, do you balance strata on that variable, treat that variable as one element of structural bias and therefore part of the independent variable, or use it as a covariate? If poverty is a big risk factor for serious parent-to-child violence in one group, but not the other, do you weight poverty the same or differently across groups? Lastly, there may be a base-rate complexity to this question. For example, if true maltreatment base rates are high in group A, and lower in group B, then a really sharp worker actually should evaluate objective risk indicators differently across the two groups (i.e. the sensitivity and specificity of the risk indicators will not be equal between groups). This might need to be factored into determining what the "correct" decision in a case should have been, because it may or may not be equivalent across groups. For example, there is a higher prevalence of sex abuse perpetration among men than among women. Therefore, we might evaluate exactly the same allegation against a mother as less probable than we might if it was against a father. One could (and many of the father advocates in the world would) call this bias or unfairness. Others might just call it objective Bayesian reasoning. To my knowledge, this issue has never been incorporated into bias measurement or disparity research, including in the vignette studies. All of this means that measuring racial bias in child welfare decision making is a complex issue--in some ways more analogous to messy quasi-experiments than to psychometrics or sociometrics. The reality of racial bias, historically or currently, is not what is in question here, and by pointing out these estimation issues my intent is not to encourage either racism denial or racism exaggeration. Rather, the issue is the accuracy and precision of our estimates, and I think we have a ways to go to measure this precisely. Mark Chaffin