Child-Maltreatment-Research-L (CMRL) List Serve

Message ID: 10582

Date: 2020-01-20

Author:Kurt Heisler

Subject:Re: Child centered outcome measures

As Rosanbalm, et al. (2016) point out (see the article attached previously), "It is important to note that the CANS was not developed using a psychometric approach ...". The CANS was developed using an approach called communimetrics, which emphasizes reliability and validity at the item level; the use of scales and domains is possible but not a critical feature of communimetric measures. (These measures are also designed to improve communication, e.g., between the provider and client, in the context of service delivery, which is something communimetric advocates believe single items can do well.) Therefore, researchers or practitioners who are trained in psychometric theory will be discouraged when trying to evaluate the use of communimetric measures through the lens of psychometrics. For example, that idea that an individual item can be more reliable and valid than a scale (i.e., multiple items) is difficult to understand, because scales and domains evolved due to limitations in measuring constructs with a single-item. However, there is some research to suggest that such limitations can be overcome. This is not an endorsement or refutation of communimetric measures. I'm simply providing some context to advance the discussion. I also would recommend reading the Rosanbalm (2016) article mentioned previously (particularly, p. 6, section 3.2.4. [CANS] Psychometric properties). It cites several studies that examined CANS for inter-rater reliability, internal consistency, construct validity (2 studies), convergent validity (evidence is weak), and predictive validity. I haven't read those studies so I am curious as to whether all of these studies used multi-item CANS domains (in which case these psychometric properties make sense) or whether some studies evaluated these properties with single items. Because the CANS emphasizes the utility of single items to support service delivery and communication, this suggests practitioners can adapt the instrument according to their service environment, in which case I'm unsure how the findings from these studies can be generalized. Best regards, Kurt Kurt Heisler, Ph.D., M.P.H., M.S. Kurt Heisler Consulting, LLC Data analytics | Data visualization | Child welfare | Agile Transformation On Mon, Jan 20, 2020 at 10:05 AM Jonathan Gould > wrote: Jessica: My question may be perceived as a thorn in one’s side but I am curious about your perspective that the CANS and CANS Trauma are wonderful decision-making tools when, if I understood your email correctly, there are questions about its construct validity. How can a data gathering tool be useful if empirical data indicates little construct validation for the factors being assessed? Said differently, as a psychologist, I have an ethical obligation to employ measures with psychometric integrity (See APA’s Ethical Standard 9). How does one argue to a licensing board that the use of a measure of unknown reliability and validity is an ethical decision? [In your email, one could argue that the measures are of poor validity based on empirical examination.] Jon Jonathan Gould, Ph.D., ABPP Diplomate in Forensic Psychology Specializing in Forensic Psychological Consultations and Evaluations 704 641 7990 (cell) 704 837 2969 (fax) jwgould53@gmail.com On Jan 18, 2020, at 12:14 PM, Jessica Bartlett > wrote: I am in agreement with my colleagues on this one. We tested both the CANS and CANS Trauma against well-validated outcome measures (e.g., UCLA-PTSD-RI; CBCL) in the Massachusetts Child Trauma Project and found troubling inconsistencies in children’s outcomes for comparable constructs. It’s a wonderful clinical decision-making tool but really needs more extensive testing in relation to assessing child outcomes.. Jess Jessica Dym Bartlett, MSW, PhD Co-Director, Early Childhood Research Director, Massachusetts Office Child Trends 460 Totten Pond Road, Suite 260 | Waltham, MA 02451 Corporate Office: 7315 Wisconsin Ave, Ste 1200W | Bethesda, MD 20814 (240) 223-9218 |jbartlett@childtrends.org On Jan 18, 2020, at 10:27 AM, Joshua P Mersky > wrote: I agree with Joe. The CANS is used here in Wisconsin. While it has value as a practice tool, it remains unproven as a scientific assessment. Josh Joshua P. Mersky Professor, Helen Bader School of Social Welfare Co-Director, Institute for Child and Family Well-being http://uwm.edu/icfw/ University of Wisconsin-Milwaukee 2400 E. Hartford Ave., Milwaukee, WI 53211 ________________________________ From: bounce-124285394-6841007@list.cornell.edu > on behalf of Joseph Ryan > Sent: Friday, January 17, 2020 3:17:36 PM To: Child Maltreatment Researcher List (CMRL) > Subject: Re: Child centered outcome measures Despite widespread use, I do not believe CANS was designed to measure change over time. Joe On Fri, Jan 17, 2020 at 9:51 AM Chen, Lijun > wrote: Hi Rich, I wonder whether you know the Child and Adolescent Needs and Strengths (CANS) assessment tool developed by John Lyons. John used to work as a colleague at Chapin Hall, now with University of Kentucky. CANS has been used in some states as an assessment tool for children being investigated or to be placed in out of home care. It has modules for different domains of child well-being and for different age groups. You can contact John to learn more about CANS. I hope this is helpful for your efforts. Lijun Chen, Ph.D. Senior Researcher Chapin Hall at the University of Chicago 1313 East 60th St., Chicago, IL 60637 Office Tel.: 773 2565140 www.chapinhall.org [Chapin Hall Logo for Signature] From: bounce-124262866-9885936@list.cornell.edu [mailto:bounce-124262866-9885936@list.cornell.edu] On Behalf Of Richard Sent: Friday, January 10, 2020 1:33 PM To: child-maltreatment-research-l@list.cornell.edu Subject: Child centered outcome measures Greetings Everyone, We are starting a conversation in Minnesota about measuring outcomes at the child level. We have in mind to get baseline measures when children come into the system, using existing validated instruments, that would include an assessment of trauma, physical and mental development, and behavioral and mental health. The idea is that over time, if the system is doing its job well, all of these metrics will improve. We recognize this is not a small undertaking but we want to get started thinking about it. The hypothesis is that by rolling up these measures, de-identified of course, at the worker, supervisor, unit, County and state levels, it will become more clear how well children are doing. Also, following the principle that “what gets measured is what it’s done”, measuring child-centered outcomes would focus on child well-being in ways that add an important dimension to the current (though also important) process-oriented child welfare metrics such as timeliness of response, frequency of social worker visits, length of time in care etc. Is anyone aware of efforts underway to measure child-level outcomes in any counties or states? Thanks for whatever help you can provide. Rich Gehrman Executive Director Safe Passage for Children of Minnesota 651-303-3209 www.safepassagemn.org [cid:image001.jpg@01D30081.29BFC050] [http://safepassagemn.com/images/twitter.png] [cid:image002.gif@01D30081.29BFC050] -- Joseph P. Ryan, Ph.D. Professor Director Child and Adolescent Data Lab ssw-datalab.org

As Rosanbalm, et al. (2016) point out (see the article attached previously), "It is important to note that the CANS was not developed using a psychometric approach ...". The CANS was developed using an approach called communimetrics, which emphasizes reliability and validity at the item level; the use of scales and domains is possible but not a critical feature of communimetric measures. (These measures are also designed to improve communication, e.g., between the provider and client, in the context of service delivery, which is something communimetric advocates believe single items can do well.) Therefore, researchers or practitioners who are trained in psychometric theory will be discouraged when trying to evaluate the use of communimetric measures through the lens of psychometrics. For example, that idea that an individual item can be more reliable and valid than a scale (i.e., multiple items) is difficult to understand, because scales and domains evolved due to limitations in measuring constructs with a single-item. However, there is some research to suggest that such limitations can be overcome. This is not an endorsement or refutation of communimetric measures. I'm simply providing some context to advance the discussion. I also would recommend reading the Rosanbalm (2016) article mentioned previously (particularly, p. 6, section 3.2.4. [CANS] Psychometric properties). It cites several studies that examined CANS for inter-rater reliability, internal consistency, construct validity (2 studies), convergent validity (evidence is weak), and predictive validity. I haven't read those studies so I am curious as to whether all of these studies used multi-item CANS domains (in which case these psychometric properties make sense) or whether some studies evaluated these properties with single items. Because the CANS emphasizes the utility of single items to support service delivery and communication, this suggests practitioners can adapt the instrument according to their service environment, in which case I'm unsure how the findings from these studies can be generalized. Best regards, Kurt Kurt Heisler, Ph.D., M.P.H., M.S. Kurt Heisler Consulting, LLC Data analytics | Data visualization | Child welfare | Agile Transformation On Mon, Jan 20, 2020 at 10:05 AM Jonathan Gould > wrote: Jessica: My question may be perceived as a thorn in one’s side but I am curious about your perspective that the CANS and CANS Trauma are wonderful decision-making tools when, if I understood your email correctly, there are questions about its construct validity. How can a data gathering tool be useful if empirical data indicates little construct validation for the factors being assessed? Said differently, as a psychologist, I have an ethical obligation to employ measures with psychometric integrity (See APA’s Ethical Standard 9). How does one argue to a licensing board that the use of a measure of unknown reliability and validity is an ethical decision? [In your email, one could argue that the measures are of poor validity based on empirical examination.] Jon Jonathan Gould, Ph.D., ABPP Diplomate in Forensic Psychology Specializing in Forensic Psychological Consultations and Evaluations 704 641 7990 (cell) 704 837 2969 (fax) jwgould53gmail.com On Jan 18, 2020, at 12:14 PM, Jessica Bartlett > wrote: I am in agreement with my colleagues on this one. We tested both the CANS and CANS Trauma against well-validated outcome measures (e.g., UCLA-PTSD-RI; CBCL) in the Massachusetts Child Trauma Project and found troubling inconsistencies in children’s outcomes for comparable constructs. It’s a wonderful clinical decision-making tool but really needs more extensive testing in relation to assessing child outcomes.. Jess Jessica Dym Bartlett, MSW, PhD Co-Director, Early Childhood Research Director, Massachusetts Office Child Trends 460 Totten Pond Road, Suite 260 | Waltham, MA 02451 Corporate Office: 7315 Wisconsin Ave, Ste 1200W | Bethesda, MD 20814 (240) 223-9218 |jbartlettchildtrends.org On Jan 18, 2020, at 10:27 AM, Joshua P Mersky > wrote: I agree with Joe. The CANS is used here in Wisconsin. While it has value as a practice tool, it remains unproven as a scientific assessment. Josh Joshua P. Mersky Professor, Helen Bader School of Social Welfare Co-Director, Institute for Child and Family Well-being http://uwm.edu/icfw/ University of Wisconsin-Milwaukee 2400 E. Hartford Ave., Milwaukee, WI 53211 ________________________________ From: bounce-124285394-6841007list.cornell.edu > on behalf of Joseph Ryan > Sent: Friday, January 17, 2020 3:17:36 PM To: Child Maltreatment Researcher List (CMRL) > Subject: Re: Child centered outcome measures Despite widespread use, I do not believe CANS was designed to measure change over time. Joe On Fri, Jan 17, 2020 at 9:51 AM Chen, Lijun > wrote: Hi Rich, I wonder whether you know the Child and Adolescent Needs and Strengths (CANS) assessment tool developed by John Lyons. John used to work as a colleague at Chapin Hall, now with University of Kentucky. CANS has been used in some states as an assessment tool for children being investigated or to be placed in out of home care. It has modules for different domains of child well-being and for different age groups. You can contact John to learn more about CANS. I hope this is helpful for your efforts. Lijun Chen, Ph.D. Senior Researcher Chapin Hall at the University of Chicago 1313 East 60th St., Chicago, IL 60637 Office Tel.: 773 2565140 www.chapinhall.org [Chapin Hall Logo for Signature] From: bounce-124262866-9885936list.cornell.edu [mailto:bounce-124262866-9885936list.cornell.edu] On Behalf Of Richard Sent: Friday, January 10, 2020 1:33 PM To: child-maltreatment-research-llist.cornell.edu Subject: Child centered outcome measures Greetings Everyone, We are starting a conversation in Minnesota about measuring outcomes at the child level. We have in mind to get baseline measures when children come into the system, using existing validated instruments, that would include an assessment of trauma, physical and mental development, and behavioral and mental health. The idea is that over time, if the system is doing its job well, all of these metrics will improve. We recognize this is not a small undertaking but we want to get started thinking about it. The hypothesis is that by rolling up these measures, de-identified of course, at the worker, supervisor, unit, County and state levels, it will become more clear how well children are doing. Also, following the principle that “what gets measured is what it’s done”, measuring child-centered outcomes would focus on child well-being in ways that add an important dimension to the current (though also important) process-oriented child welfare metrics such as timeliness of response, frequency of social worker visits, length of time in care etc. Is anyone aware of efforts underway to measure child-level outcomes in any counties or states? Thanks for whatever help you can provide. Rich Gehrman Executive Director Safe Passage for Children of Minnesota 651-303-3209 www.safepassagemn.org [cid:image001.jpg01D30081.29BFC050] [http://safepassagemn.com/images/twitter.png] [cid:image002.gif01D30081.29BFC050] -- Joseph P. Ryan, Ph.D. Professor Director Child and Adolescent Data Lab ssw-datalab.org