This Expert Panel is Archived.

While this Expert Panel is no longer active, we invite you to review and recommend past replies and resources. Membership for this Expert Panel is closed, but we hope you'll join us in one of the many communities on GHDonline.

Panelists of Strengthening and Scaling the Community Health Workforce and GHDonline staff

How do you measure the effectiveness and impact of CHW programs?

Posted: 11 Jun, 2017   Recommendation: 1   Replies: 13

What data should you collect? From where, by whom, and how often?

Replies

 

Isha Nirola Panelist Replied at 8:12 AM, 12 Jun 2017

One of the greatest values of many CHW programs, depending on the design and area of focus, is the formation of a decentralized community based care delivery system that can capture even the most vulnerable populations. This type of system can have huge success at alleviating human suffering at a population health level. At Possible, we have worked across teams to develop an M&E system for population health impact using the following process:

1) Program leads to identify area(s) of focus. This could be done by doing an assessment on the burden of disease in our areas of coverage, or by aligning with national priorities as identified by the Ministry of Health. At Possible we have selected our CHW program areas of focus in partnerships with the Ministry of Health in Nepal - Reproductive, Maternal, and Child Health as well as Non-Communicable Diseases.
2) Understanding what primary and secondary indicators we should use to measure the success and impact of the CHW program interventions. I've attached a process that was created by our inimitable Impact Team leadership, led by Dr. David Citrin and Mr. Poshan Thapa. This document, that internally we refer to as Possible's Metrics Creation Protocol, lays out a step by step process for program directors to develop metrics and targets.
3) Once metrics are created, our Impact team, in partnership with the Program leadership, sets out to develop data collection forms, that are also sometime simply point-of-care patient forms, that CHWs use to both collect data and deliver care. The questions on these forms are used to get the information for the numerator and denominators of the metrics that have been determined. While we measure data at different frequencies, data collection occurs on a monthly basis, and majority of our program indicators are reported monthly, but some are reported quarterly or annually, depending on the program need or audience.
4) Once monthly data collection is complete, there needs to be a system that feeds data back to program managers. Our Impact team reports our indicators on our Monthly Impact Dashboard, but also provides datasets and summaries back to the programs teams on a monthly basis.
5) Lastly, we have systematized the accountability for utilization of data through what we call the CHW Internal Data Review. This is a system in which every tier of our Community Healthcare Team structure, including the CHWs, look at the metrics for their area to understand program gaps and geographic and demographic information to identify populations where we have not reached our targets. Additionally, each tier is expected to propose an action plan for the coming month(s) on how they intend to reach their targets, or at least move the needle in that direction.

While M&E and QI of many other CHW systems globally may play out differently, Possible's approach is one way to measure impact with the ultimate goal of increasing access to care and alleviating human suffering at a population health level. The most important thing is that a structure is in which these components are considered so that a strong M&E system is in place.

Attached resource:

Tess Panizales, DNP, MSN, RN Replied at 2:22 PM, 12 Jun 2017

Couple of years back, I did a paper on CHW for one of my doctoral program class on Global Health. What was notable is the role variation among CHW or its other derivatives (it can be called differently across nations and programs). The role can then be further redefined by the hiring organization. In general, the metrics to measure the impact of CHW's is organization or program based.
Is this effective, given the greater good CHW brings into healthcare across nations? Can we look at a common standard in training and roles to help identify a global metric for our CHWs? These were the underpinning issues on my class paper. Wearing my quality hat, I looked at the structures and the processes that CHWs are engaged and outcomes to be measured. The paradigm I designed look at culture, education and skills training. By creating a common standard, a database can then be develop as repository. Variables on competency, evaluation and outcome measures needs to define the data collection tool. I believe this can be done through a collective effort and given the advance ICT we now have. We definitely know CHWs are our partner in care.

CYRIANUS LWEYENDEZA Replied at 3:23 PM, 12 Jun 2017

This is great, thank you!

Omur Cinar Elci Replied at 3:32 PM, 12 Jun 2017

Tess, thank you for sharing this information. It sounds quite interesting; do you mind sharing little more details or the copy of your paper? Thank you. Regards, Omur

Mei Lin Fung Replied at 4:52 PM, 12 Jun 2017

From our point of view where we work on how people can Connect to Thrive,
the PeopleCentered.net works closely with 100Mlives.org and we use the
measurement framework (http://www.100mlives.org/wp-content/uploads/2016/04/IHI_100MHL_Metrics_Spring...) so communities can not only measure progress but compare and learn and take advantage of each other's break throughs.

100Mlives.org is a global expansion of work that was catalyzed by
www.IHI.org with the US Dept of Health and Human Services - where networks of communities came together to develop Breakthrough Collaboratives on topics of interest.

Here's a write up from 2012. We are working on an updated version.

Trail Guides and Trading Posts on the Journey to Health:
https://drive.google.com/file/d/0BxWhDjUmHhD6RG9hbmhkc1AwVWc/view?usp=drive_web

Mei Lin Fung
Organizer, People Centered Internet, co-founded with Vint Cerf
http://www.peoplecentered.net/
External advisor to the Stanford Center for Population Health Science: http://med.stanford.edu/phs.html
Member of the Global Future Council on Digital Economy and Society: https://www.weforum.org/communities/the-future-of-the-digital-economy-and-soc...
Member of the Steering Committee, World Economic Forum, Internet for All: https://www.weforum.org/projects/internet-for-all
Vice Chair, Internet Inclusion, IEEE Internet Initiative: http://internetinitiative.ieee.org/

Jim Lloyd Panelist Replied at 9:23 AM, 13 Jun 2017

To everyone in this thread:

Thanks for the great discussion! We and the organizations with which we work are VERY interested in these questions and we will be following along closely.

Anna Heard Replied at 9:38 PM, 13 Jun 2017

While some very good points have been made, measuring effectiveness can
only be done using a counterfactual - a comparison group that represents
what would have happened without the CHWs. And then also or later, you can
compare to an alternative intervention to see which is more effective. But
monitoring won't really tell you whether your CHWs are making things better
(or not) or if something else is happening at the same time.
So, it depends on your definition of effectiveness and impact on how to
measure "effectiveness and impact."
But key is identifying appropriate indicators and finding ways to get that
data reliably and accurately.
Maybe that want really the question...

Isha Nirola Panelist Replied at 1:07 AM, 14 Jun 2017

Thanks for the great comments and for sharing additional resources. I see this platform as an important forum to do this.

I wanted to respond to Tess' comments above. In response to your question "Can we look at a common standard in training and roles to help identify a global metric for our CHWs?" my thinking is a general indicator for CHWs may be difficult to develop. The reason I say this is because globally, CHW programs vary in scope of work. For example, some CHW programs are focused on preventative care only. Others might be more care delivery focused (ie - treatment of diseases), and others may focus on referrals to health facilities - more of a social worker role. Because the term CHW encapsulates so many different types of health workers who provide community based care. However, I do think there are process measures that can be put in place to assess the effectiveness of CHWs in a particular program. One of our indicators looks at the percentage of patients homes a CHW has reached in a particular month. An indicator like this is less about the population health impact the program is making, and focused more around performance of CHWs. So I do think there are CHW-specific performance metrics that can and should be developed, as opposed to larger program or population health metrics.

Celia Taylor Replied at 3:32 PM, 14 Jun 2017

One step on the pathway to CHW programme effectiveness is a cadre of safe, knowledgeable and skilled CHWs (in my boss' language, this would be a necessary but not sufficient condition). So an intermediate outcome measure (even before field-based performance) would be CHWs' post-training knowledge and skills about the role/tasks they have been recruited for. iCCM and other comprehensive CHW programmes cover a range of often-overlapping disease/health areas (malaria, deworming, malnutrition etc.), for which international standards exist in at least some cases. Notwithstanding the lack of complete uniformity within a disease/health area across programmes, would there be interest in a series of "off the shelf" post-training assessment tools (written and/or practical tests) for different disease/health areas that could be used across programmes? This would have a number of advantages, including having access to a test bank that has been rigorously developed, so valid and reliable tests can be constructed at no cost to a programme, being able to provide CHWs with feedback on their performance to maximize educational impact and potentially the ability to compare different approaches to training using a robust outcome measure.

Ochiawunma Ibe Replied at 6:19 PM, 14 Jun 2017

In response to Tess's comments and Isha's inputs I would like to highlight that World Health Organization is in the process of developing guidelines to assist national governments, as well as national and international partners to improve the design, implementation, performance and evaluation for CHW programs. They have been surveying various stakeholders to better understand the views of stakeholders in relation to the feasibility and acceptability of some of the interventions and recommendations under consideration, and the values assigned to the outcomes of the interventions. Their overall aim is to optimize the design of the guidelines and maximize their uptake and use by planners and policy makers in countries. Participants who attended the Institutionalizing Community Health Conference held in South Africa from March 27-30, 2017 were invited to respond to the survey

Giorgio Cometto Replied at 10:30 AM, 15 Jun 2017

thanks everyone for a very informative discussion so far. I would like indeed to confirm that at WHO we are dedicating significant time and attention to evidence and normative guidance to optimize the performance of CHWs. The ongoing guideline development process http://who.int/hrh/community/en/ as part of the initial conversation has identified outcome measures to assess the effectiveness of CHW interventions, and we have decided to adopt (and adapt) Naimoli's logic model https://human-resources-health.biomedcentral.com/articles/10.1186/1478-4491-1... , which identifies a comprehensive strategy to measure the performance of CHW programmes at output, outcome and impact level.

In response to Ochiawunma's comment, thanks for flagging our participation at the ICHC in South Africa. As a matter of fact, we have decided to extend the stakeholder perception survey to assess the feasibility and acceptability of the emerging recommendations of the forthcoming guidelines. You are all welcome to participate in the survey, which you will find here http://who.int/hrh/community/better-understand-of-views/en/

Kenneth Maes Replied at 4:17 PM, 16 Jun 2017

Hi everyone! Sorry to be late to this discussion. I would just like to share with you a brief overview of the CHW Common Indicators Project that begun a couple years ago with the Michigan Community Health Worker Alliance and has since grown to include nearly 50 people communicating across 10 US states. The main aim of the CHW Common Indicators project is to develop, recommend, and help scale up the use of a list of process and outcome metrics that every CHW program in the U.S. should use to evaluate CHW impacts at multiple levels (individual, CHW, program, population, policy). The Project is ongoing and still has a lot of work to do, but we have developed a consensus on a list of constructs to measure, and have begun pulling together resources to know how/if they are already being measured by folks in the US (and globally). For more information, visit this website:
http://www.michwa.org/common-indicators-project-2/

Or, please contact me (), Leticia Rodriguez-Garcia (), and/or Noelle Wiggins ().

Thank you!

Kenny Maes
http://liberalarts.oregonstate.edu/users/kenneth-maes

Isha Nirola Panelist Replied at 6:34 PM, 16 Jun 2017

Thanks Kenneth!! Great resource!

This Expert Panel is Archived.

While this Expert Panel is no longer active, we invite you to review and recommend past replies and resources. Membership for this Expert Panel is closed, but we hope you'll join us in one of the many communities on GHDonline.

Panelists of Strengthening and Scaling the Community Health Workforce and GHDonline staff