Child-Maltreatment-Research-L (CMRL) List Serve

Message ID: 10712

Date: 2019-02-19

Author:Mackert - CDHS, Marc

Subject:Re: CQI programs in child welfare

Rich, Colorado also has a qualitative case review process. Another reply indicated some of the challenges with qualitative case reviews (sample size, consistency across reviewers, etc.). While there are challenges, we have found many of these can be minimized, and the reviews themselves can have a significant impact on outcomes for children. For example, in a study that was conducted on our periodic administrative review process (completed once every six months that a child is placed in out-of-home care) it was found that, when we reviewed the children/youth timely, they achieved permanency approximately 11 months sooner than those who we did not review timely. This has a positive impact on outcomes for children/youth, and also generates cost savings for the state (as the cost of out-of-home placement is reduced by those same number of months, for every child reviewed timely). Here is a citation to the article: Whitaker, T.M. (2011). Administrative case reviews: Improving outcomes for children in out-of-home care. Children and Youth Services Review, 33, 1683-1708. Tia still works for us, so if you were interested in a copy of the article, if you email me, Tia could make a copy available. Similar to what you described in regards to Utah, our staff meet with the caseworkers to go over the findings, we aggregate the findings to create county level reports that are discussed with county administration. For the administrative reviews (foster care reviews), we also compile quarterly reports. We also discuss the findings with our Division of Child Welfare to help inform their knowledge of county level performance, as well as statewide performance. We have a logic model that we use to show how we believe the focus of our reviews on certain child welfare practices can be part creating improved outcomes for the children and families served through Colorado's child welfare system. Additionally, we have other qualitative review processes that have led to significant improvements in Colorado's child welfare system. For example, we conduct an annual review of screened out referrals. We have found improvement at the statewide level in the accuracy of screening decisions over years we have conducted the review (we have held 11 reviews over 12 years). We published an article regarding that through the American Humane Association. We also helped facilitate a process that was instrumental in helping create Colorado's current safety and risk assessment tools. We published an article regarding the process, and you can find that here: Orsi, R., Drury, I., and Mackert, M. (2014). Reliable and valid: A procedure for establishing item-level interrater reliability for child maltreatment risk and safety assessment.Children and Youth Services Review, 43, 58-66. If you were interested, and don't have access, if you email me, I can share a copy of that article with you as well. Both qualitative case review processes, and more quantitative processes (like a stat program), require care in understanding their strengths and limitations. As an example, stat programs are also beholden to the data integrity/accuracy that they are based on. In essence, any CQI process needs an honest assessment of it's inherent strengths and limitations, so that well informed and appropriate decisions can be made in light of those. A best case scenario is likely having both types of processes. A quantitative process for monitoring outcomes, and perhaps other activities that can be easily measured through child welfare information systems/databases, as well as qualitative case reviews that focus on evaluative aspects of child welfare practice. In particular, those practices you believe most closely aligned with the outcomes you want to improve. Qualitatively monitoring if, and how well, practices are being implemented (and perhaps fidelity to models as well), and whether they are having the desired impact on outcomes, can be critical to informing later program/policy changes or initiatives. You can learn more about our process here: https://www.colorado.gov/pacific/cdhs/administrative-review The instruments that we use across the various review populations are available on the "Qualitative Case Reviews" link. We conduct reviews of: * Calls to our child welfare hotline * Screened out referrals * Referrals accepted for assessment * Cases receiving in-home services * Foster care reviews (six-month, periodic reviews, also known as administrative reviews) * Foster home certification reviews * Reviews of fatal, near fatal, and egregious incidents * Reviews of adult protective services cases If you have any questions regarding our process, please feel free to give me a call. I'd be happy to talk with you more about any of our processes. My information should show in my signature line, but here it is as well: 303-866-7137. Thanks! Marc On Mon, Feb 18, 2019 at 7:04 AM Richard > wrote: Greetings Everyone, We have been asked by a Minnesota legislator whether there are any well-developed Continuous Quality Improvement (CQI) programs child welfare programs in other states for possible inclusion in a bill. Also, it would be helpful to know about appropriate programs in related fields. She is looking for an approach that is less complicated and more a part of daily operations than the Child and Family Service Reviews. The closest one I am aware of is the Utah Case Process Review program. It reviews key process measures such as monthly visits by caseworkers, and key quality measures such as whether case plans have been implemented and mental health needs addressed. This legislator is also looking for a protocol that collects child outcome metrics such as whether the child’s involvement with child protection has reduced trauma, improved cognitive and physical development, and improved behavioral or mental health challenges. I don’t know if it is still the case, but at one point the Utah program was implemented at every level. Caseworkers did periodic peer reviews, supervisors review their staffs’ cases on a regular basis, and the results were rolled up quarterly reports. That would be an very desirable feature for anything proposed in legislation here. Thanks for any help you can give us on this topic! Rich Gehrman Executive Director Safe Passage for Children of Minnesota 651-303-3209 www.safepassagemn.com -- Marc J. Mackert, Ph.D. Director P: 303-866-7160 l Fax: 303-866-7658 l Cell: 720-512-8814 4045 South Lowell Boulevard, Denver CO 80236 marc.mackert@state.co.us l https://www.colorado.gov/pacific/cdhs/administrative-review Under the Colorado Open Records Act (CORA), all messages sent by or to me on this state-owned e-mail account may be subject to public disclosure. This email and any attachments, from the Colorado Department of Human Services, are confidential and intended solely for the use of the individual or entity to which it is addressed. The information contained herein may include protected or otherwise privileged information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you have received this message in error, please notify the sender by reply to this message and delete the email without disclosure.

Rich, Colorado also has a qualitative case review process. Another reply indicated some of the challenges with qualitative case reviews (sample size, consistency across reviewers, etc.). While there are challenges, we have found many of these can be minimized, and the reviews themselves can have a significant impact on outcomes for children. For example, in a study that was conducted on our periodic administrative review process (completed once every six months that a child is placed in out-of-home care) it was found that, when we reviewed the children/youth timely, they achieved permanency approximately 11 months sooner than those who we did not review timely. This has a positive impact on outcomes for children/youth, and also generates cost savings for the state (as the cost of out-of-home placement is reduced by those same number of months, for every child reviewed timely). Here is a citation to the article: Whitaker, T.M. (2011). Administrative case reviews: Improving outcomes for children in out-of-home care. Children and Youth Services Review, 33, 1683-1708. Tia still works for us, so if you were interested in a copy of the article, if you email me, Tia could make a copy available. Similar to what you described in regards to Utah, our staff meet with the caseworkers to go over the findings, we aggregate the findings to create county level reports that are discussed with county administration. For the administrative reviews (foster care reviews), we also compile quarterly reports. We also discuss the findings with our Division of Child Welfare to help inform their knowledge of county level performance, as well as statewide performance. We have a logic model that we use to show how we believe the focus of our reviews on certain child welfare practices can be part creating improved outcomes for the children and families served through Colorado's child welfare system. Additionally, we have other qualitative review processes that have led to significant improvements in Colorado's child welfare system. For example, we conduct an annual review of screened out referrals. We have found improvement at the statewide level in the accuracy of screening decisions over years we have conducted the review (we have held 11 reviews over 12 years). We published an article regarding that through the American Humane Association. We also helped facilitate a process that was instrumental in helping create Colorado's current safety and risk assessment tools. We published an article regarding the process, and you can find that here: Orsi, R., Drury, I., and Mackert, M. (2014). Reliable and valid: A procedure for establishing item-level interrater reliability for child maltreatment risk and safety assessment.Children and Youth Services Review, 43, 58-66. If you were interested, and don't have access, if you email me, I can share a copy of that article with you as well. Both qualitative case review processes, and more quantitative processes (like a stat program), require care in understanding their strengths and limitations. As an example, stat programs are also beholden to the data integrity/accuracy that they are based on. In essence, any CQI process needs an honest assessment of it's inherent strengths and limitations, so that well informed and appropriate decisions can be made in light of those. A best case scenario is likely having both types of processes. A quantitative process for monitoring outcomes, and perhaps other activities that can be easily measured through child welfare information systems/databases, as well as qualitative case reviews that focus on evaluative aspects of child welfare practice. In particular, those practices you believe most closely aligned with the outcomes you want to improve. Qualitatively monitoring if, and how well, practices are being implemented (and perhaps fidelity to models as well), and whether they are having the desired impact on outcomes, can be critical to informing later program/policy changes or initiatives. You can learn more about our process here: https://www.colorado.gov/pacific/cdhs/administrative-review The instruments that we use across the various review populations are available on the "Qualitative Case Reviews" link. We conduct reviews of: * Calls to our child welfare hotline * Screened out referrals * Referrals accepted for assessment * Cases receiving in-home services * Foster care reviews (six-month, periodic reviews, also known as administrative reviews) * Foster home certification reviews * Reviews of fatal, near fatal, and egregious incidents * Reviews of adult protective services cases If you have any questions regarding our process, please feel free to give me a call. I'd be happy to talk with you more about any of our processes. My information should show in my signature line, but here it is as well: 303-866-7137. Thanks! Marc On Mon, Feb 18, 2019 at 7:04 AM Richard > wrote: Greetings Everyone, We have been asked by a Minnesota legislator whether there are any well-developed Continuous Quality Improvement (CQI) programs child welfare programs in other states for possible inclusion in a bill. Also, it would be helpful to know about appropriate programs in related fields. She is looking for an approach that is less complicated and more a part of daily operations than the Child and Family Service Reviews. The closest one I am aware of is the Utah Case Process Review program. It reviews key process measures such as monthly visits by caseworkers, and key quality measures such as whether case plans have been implemented and mental health needs addressed. This legislator is also looking for a protocol that collects child outcome metrics such as whether the child’s involvement with child protection has reduced trauma, improved cognitive and physical development, and improved behavioral or mental health challenges. I don’t know if it is still the case, but at one point the Utah program was implemented at every level. Caseworkers did periodic peer reviews, supervisors review their staffs’ cases on a regular basis, and the results were rolled up quarterly reports. That would be an very desirable feature for anything proposed in legislation here. Thanks for any help you can give us on this topic! Rich Gehrman Executive Director Safe Passage for Children of Minnesota 651-303-3209 www.safepassagemn.com -- Marc J. Mackert, Ph.D. Director P: 303-866-7160 l Fax: 303-866-7658 l Cell: 720-512-8814 4045 South Lowell Boulevard, Denver CO 80236 marc.mackertstate.co.us l https://www.colorado.gov/pacific/cdhs/administrative-review Under the Colorado Open Records Act (CORA), all messages sent by or to me on this state-owned e-mail account may be subject to public disclosure. This email and any attachments, from the Colorado Department of Human Services, are confidential and intended solely for the use of the individual or entity to which it is addressed. The information contained herein may include protected or otherwise privileged information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you have received this message in error, please notify the sender by reply to this message and delete the email without disclosure.