By Svetha Janumpalli, Founder and CEO 

Over the past year, you have likely heard us mention that we are growing very quickly. In fact, in 2022, we have reached more infants than in all previous years (combined) since piloting the program in late 2016. This growth, driven by new funding, was spurred by positive results from an independent randomized controlled trial (RCT) of our program and GiveWell’s analysis that found New Incentives’ program to be one of the most cost-effective ways to save a life.

As you may also know, scaling up a program while retaining impact found through an RCT is complex, but it has been our key priority. Many questions need to be considered. In our case, we have needed to understand: Is vaccination coverage similarly low in areas that we are looking to expand to? Is there enough vaccine supply to meet the growing demand we are generating? Does the population have similar (or better) attitudes about childhood vaccination? Northern Nigeria is also an insecure area, so we have to ask if the security situation is impacting the program. Above all, we need to make sure the program is implemented well: Are our staff, clinic staff, and caregivers adhering to protocols that make the program work?

In this post, I will share five things we are doing at New Incentives to ensure we continue to make a substantial impact – in line with the results from the RCT – as we scale up. It is a process we call data-driven scaling, which involves daily data collection, analysis, and processes for making decisions based on that data.
‍

1. We collect and use data on vaccination coverage.

The RCT of our program at the demonstration stage indicated the program worked well in a context where vaccine coverage was quite low and the child mortality rate was high. This was consistent with the RCT results of comparable interventions. If vaccine coverage is already quite high, our assumption is that the program may not be as effective. 

Therefore, to identify new regions and geographies to expand to, we look at government and global health data on childhood mortality, disease burden, vaccination demand, and vaccination supply by state. 

In addition, without randomization (which isn’t possible as we scale), we need a way to reliably estimate whether our program is boosting childhood vaccination rates in the areas where we work. Using a protocol designed by an independent research firm, we collect ongoing data on coverage of routine childhood vaccinations. Surveyors carry out household surveys before going into an area (to measure baseline vaccination rates), then every six months to one year to assess to what extent the program has increased vaccination rates. In late 2022, New Incentives will also measure vaccination coverage in areas where our program does not operate. This data will allow us to estimate the program’s impact on coverage rates at scale. However, these areas are not identical to the program areas (as they would be with an RCT), so we interpret the data with caution.

‍‍

2. We track vaccine supply.

Our program increases demand for vaccinations, and sufficient supply is needed to meet this increased demand. The RCT showed results in clinics with minimum adequate vaccination services (clinic staff, related commodities, etc.)and vaccine supply. Changes to key assumptions like the availability of vaccines, condition of vaccines, or presence of vaccination services and clinic staff could mean the target population is changing or assumptions for the theory of change to work are not present, affecting our impact.

We collect real-time supply data from the ground level and use this data to take actions. We use data from clinics and stakeholders to identify issues; increase coordination up the supply chain from clinics up to “apex clinics” to the local government area (LGA), the state, zonal, and national levels to ensure vaccines reach clinics; gather information while working at clinics; and share this data at each step.

‍‍

3. We gather and use population data.

If a population has different characteristics – for example, more geographic dispersion – the program may not work as well or require adaptation. To understand population trends, we collect and use data from paper records and drawn maps from clinics, satellite and GIS population data based on clinic locations, and different radial estimates.

We use this data for constant monitoring, comparing program enrollments to population targets based on different population datasets. We identify areas where uptake is low and then work in collaboration with clinics, stakeholders, and partners to target awareness activities and outreach to those areas. This data is collected from and shared with communities and clinics to drive awareness activities and carry out targeted outreach, such as mobile vaccination sessions in select areas. 

‍‍

4. We mitigate security risks through continual monitoring and training.

The security situation is very fragile in the areas where we work, and a worsening security situation will likely dampen the program’s impacts. While security issues are out of our control, we can monitor the security situation to mitigate risks to our staff and to understand the impact it has on the program.

We do this by:

• Conducting regular security training for staff to develop a culture of security awareness
• Reviewing and updating clinic and settlement security assessments where the program operates at least once every quarter
• Making security risk level an integral component of decisions regarding expansion to states/LGAs – we do this by conducting security assessment of locations planned for expansion
• Issuing weekly security updates
• Diligently tracking and mitigating any security incidents involving staff
• Continuously sourcing security incidents affecting communities and travel routes
• Updating security status and related procedures at the end of each week
• Improving our preparedness for incident management by building the capacity of Incident Management Team (IMT) members

‍

5. We carefully monitor implementation to prevent fraud and ensure quality.

With any program that gives out cash, close monitoring is crucial. We monitor our program by collecting and reviewing data on a comprehensive set of indicators pertaining to fraud and staff supervision, enrollments, and cash disbursements. Here’s a sampling of the ways we actively work to prevent fraud and ensure quality: 

• Taking photographs of every disbursement to review the amount of cash disbursed and triangulating that with other data sources 
• Using biometrics to prevent and identify cases of caregivers enrolling the same infant multiple times
• Performing unannounced visits by managers and internal auditors for 15-20% of disbursement days
• Analyzing metadata and trends to identify potential abnormalities and loopholes
• Checking for old BCG scars before enrolling a new infant (BCG, or bacille Calmette-Guerin, is a vaccine for tuberculosis disease)
• Consulting the clinic’s child vaccine card before enrolling a new infant
• Checking administrative records to verify vaccinations before disbursing cash incentives

In addition to monitoring cash disbursements, we analyze logistics and expenditure data to prevent fraud and ensure the program is high quality and reaches beneficiaries as intended. In a follow-up post, I will talk about implementation problems we have identified and how we are fixing them. 

Importantly, as we scale, we also have to scale up our systems for monitoring and using data to drive decision-making. To do this, we are developing systems and creating apps using artificial intelligence that provide real-time insights and are easy to use and scale. 

‍

As we continue to grow, our systems will evolve as we learn. What will remain the same is our vigilant approach to using data to understand what works and what doesn’t, so we can fulfill our mission to save lives in a manner that does the most good per dollar spent. 

Watch this space for more updates. 

‍

‍