BID Initiative Lessons Learned Encyclopediabidinitiative.org/wp-content/uploads/FINAL_Lessons...DDUM district data use mentor EIR electronic immunization registry EMM enterprise mobility

June 2018

BID Initiative Team | PATH

BID Initiative

Lessons Learned

Encyclopedia

June 2018 BID Initiatives Lessons Learned Encyclopedia

ii

Background

Routine immunizations and new vaccine introductions are two of the best buys in global health.

However, a major obstacle to improving immunization coverage and optimizing service delivery

is that decision-makers at all levels of the health system have not had access to high-quality

data to use in effectively making decisions in planning, performance management, and the

delivery of services.

The Better Immunization Data (BID) Initiative is grounded in the belief that better data, plus

better decisions, will lead to better health outcomes. It was designed in partnership with the

governments of Tanzania and Zambia to enhance immunization and overall health service

delivery by improving data collection, quality, and use. The BID Initiative developed, tested, and

rolled out interventions that address some of the most pressing routine immunization service

delivery challenges, such as poor visibility into vaccine supplies and difficulty identifying children

who default on immunization schedules.

BID took a holistic approach to address immunization data challenges by packaging together

information system products, data management policies, and evidence-based practices with

people who are empowered to improve decision-making. Additionally, through the BID Learning

Network, many of the 20 participating countries across sub-Saharan Africa were consulted

during the design and testing of tools to help ensure that solutions are relevant for and can be

adopted by other countries interested in improving their health programs through better data.

The BID Initiative had three focus areas for interventions: an electronic immunization registry,

stock management between the facility and district, and creation of a culture of data use. To

build a culture of data use, change management techniques were used to understand where

health care workers were with regard to confidence and consistency in data use, and from that

information-gathering period, several interventions were developed, including data use

campaigns and introduction of a data use guide for facilities.

This document captures the various lessons learned throughout the BID Initiative, from the

design and testing phase through implementation, noting if the lesson was learned in the

Tanzania or Zambia implementation (or both).


iii

Table of Contents

Background .................................................................................................................................. i

Abbreviations .............................................................................................................................. v

Data Quality and Use Interventions ............................................................................................. 1

User-centered design (Tanzania and Zambia) ....................................................................... 1

Job description intervention (Tanzania and Zambia) ............................................................. 1

Facility cross-site visits and WhatsApp peer groups (Tanzania and Zambia) ........................ 2

Microtraining videos (Tanzania and Zambia) ......................................................................... 4

Data visualization, dashboards, and reports (Tanzania and Zambia) .................................... 5

Data use video games (Tanzania and Zambia) ..................................................................... 7

District data use guide (Tanzania and Zambia) ..................................................................... 8

Data use posters (Tanzania and Zambia) .............................................................................. 9

Workflow redesign (Tanzania and Zambia) ........................................................................... 9

Data use best practices (Tanzania and Zambia) ................................................................. 10

Change Management................................................................................................................ 12

Change readiness assessment (Tanzania and Zambia) ...................................................... 12

Community understanding of barcodes (Tanzania and Zambia) .......................................... 13

Parallel reporting systems can inhibit adoption of new tools (Tanzania and Zambia) .......... 14

Rollout/Implementation Strategy ............................................................................................... 15

Touches strategy: Timing of facility touches and onsite or classroom-style touches

(Tanzania and Zambia) ....................................................................................................... 15

Define what makes a touch successful (Tanzania and Zambia) .......................................... 17

Data use mentors (Tanzania) .............................................................................................. 18

Training BID and district-level Ministry of Health staff to understand the interventions

through role-play exercises (Tanzania and Zambia) ............................................................ 21

Peer learning network ............................................................................................................... 22

Have a community manager in place .................................................................................. 22

Webinars ............................................................................................................................. 23

Communication medium and keeping the community active ............................................... 24

Peer network governance structure ..................................................................................... 25

Sustainability and Scale ............................................................................................................ 27

How Tanzania and Zambia are scaling................................................................................ 27

Electronic Immunization Registries ........................................................................................... 28

Unique identification of children using barcodes or QR codes (Tanzania and Zambia) ........ 28

Community birth registration (Tanzania and Zambia) .......................................................... 29

Locking devices to conserve data bundles (Tanzania and Zambia) ..................................... 31

Back-entry of data (Tanzania and Zambia) .......................................................................... 32

Procurement ....................................................................................................................... 34

Software Development Process ................................................................................................ 35

Change logs (Tanzania and Zambia) .................................................................................. 35

Defining system requirements (Tanzania and Zambia) ........................................................ 36


iv

Local support for software development is needed (Tanzania and Zambia) ........................ 37

System and software updates (Tanzania and Zambia) ........................................................ 37

Development time from three to six months or longer .......................................................... 38

Deciding on a new system when the current system does not meet requirements .............. 39


v

Abbreviations

BID Initiative Better Immunization Data Initiative

BLN BID Learning Network

CHMT Council Health Management Team

CRDM Collaborative Requirements Development Methodology

DDUM district data use mentor

EIR electronic immunization registry

EMM enterprise mobility management

HCW health care worker

QR quick response

SMS short message service

TImR Tanzania Immunization Registry

UNICEF United Nations Children’s Fund

USSD unstructured supplementary service data

ZEIR Zambia Electronic Immunization Registry


1

Data Quality and Use Interventions

User-centered design (Tanzania and Zambia)

Background: A user-centered design approach is a key strategy for the development and design

of scalable and sustainable interventions. This process was started by sharing the proposed

data quality and use interventions with and soliciting feedback from a user advisory group.

Thereafter, the various interventions were tested in designated urban, peri-urban, and rural

facilities in Tanzania and Zambia. User feedback contributed to design improvements of the

interventions to produce a final package of data quality and use interventions designated for

deployment. Through their consultation, users became part and parcel of the new product.

Solutions adopted: Not all proposed interventions were adopted, based on user input on the

design approach. All the data quality and use interventions were tested, including the electronic

immunization registries (EIRs), and only the refined recommended solutions deemed

appropriate with the user advisory group(s) and government were adopted and implemented. In

Tanzania the EIR adopted is called the Tanzania Immunization Registry or TImR and in Zambia

the EIR is called the Zambia Electronic Immunization Registry or ZEIR.

Lessons learned and challenges:

Involvement of users, especially district staff, is key. Ownership and acceptance of a new

product or intervention is greatly increased when the users are consulted. When users are

consulted, resources are not wasted by launching interventions that are not tested or

accepted by the users.

It is vital to not scale beyond the initial testing facilities if the intervention still has bugs or

challenges; roll out the interventions into a few facilities first (6-20 depending on the size of

the supporting team), before scaling into the whole district. This can mitigate issues that may

be detrimental if the users lose faith in the quality of the intervention. While health care

workers (HCWs) agree on common problems, they often disagree on common solutions,

and an intervention design has to address a compromising solution.

Recommendations: Engage users at all levels of the health care system to gain feedback for

system development and to ensure ownership and sustainability of the interventions.

Job description intervention (Tanzania and Zambia)

Background: One of the initial data quality and use interventions to be tested was an

assessment of whether and how current standard operating procedures, policies, and job

descriptions included data use as part of defined roles. The hypothesis was that if these

documents did not include data use that discussions with the government about including data


2

use as a responsibility would increase both responsibility and accountability for data use across

the health system. This documentation would clarify the roles and responsibilities within facilities

surrounding data and the information flow, both vertically (facility to district) and horizontally

(within the facility). In our assessment, we found no existing roles dedicated to immunization

data quality or data use, but that data use occurred through administrative functions such as

maintaining patient records and through periodic district supervision and feedback to HCWs at

the facility level.

Solutions adopted: In the end, this intervention was neither developed nor implemented.

Lessons learned and challenges: Based on feedback from the user advisory group(s), which

included HCWs from across the health system, the process to officially update the job

descriptions would have been too time consuming. These modifications also would have

required changes across several institutions beyond the Ministries of Health (e.g., the Ministry of

Labor and the President’s Office of Regional Administration and Local Government). Further,

we learned that due to staff shortages, many HCWs do not refer to the documentation on their

roles and responsibilities because most cover a wide range of activities in their health facility,

beyond those in their job descriptions (e.g., an HCW will work in several clinical care services

within one facility).

Recommendations:

Establish a partnership between health facility supervisors and the district so that data

review meetings are used to address data use gaps when these are not formally

documented in official employee job descriptions. These meetings and supportive

supervision can drive demand for data from the district level, and as facility HCWs aim to

meet this demand for data use and performance expectations, data quality can also

improve.

Develop data use guides. The integration and use of tools such as data use guides for the

facility can drive data use even when formal documentation of data use as a responsibility

does not exist. Data use guides can help HCWs to strive beyond simple submission of a

report to owning the process to use the data in their everyday activities.

Facility cross-site visits and WhatsApp peer groups (Tanzania and Zambia)

Background: Early literature reviews and landscape analysis demonstrated the value of peer

learning networks as a way to engage and create cross-learning. Cross-site visits were

identified as a potential intervention to provide a forum for facility-to-facility in-person exchanges

and strengthening of the peer network. The focus would be to leverage data and system user

champions between facilities to serve as influencers within the network and between facilities.

Connecting champions with other peers working toward the same goals would allow for easy

http://bidinitiative.org/wp-content/uploads/BID_toc_lit_review.pdf

http://bidinitiative.org/bid-learning-network/resources/bid-initiative-landscape-analysis-current-and-past-projects-targeting-data-quality-and-use/


3

sharing of solutions and lessons to leverage a higher success rate of adoption of the EIRs and

use of the data. This intervention would create exchanges of creative solutions to improve

service delivery in areas such as workflow efficiencies.

Solutions adopted: We tested cross-site facility visits in both Tanzania and Zambia, and while

they proved to be useful, the visits were too costly for facilities and districts to own, thus

unsustainable. Instead, peers began to connect across the health system using WhatsApp; use

of this free tool continues in both Tanzania and Zambia.

Lessons learned and challenges: Cross-site visits between facilities as part of peer networking

was discouraged due to lack of staff capacity within the Ministries of Health, limited time frames,

and unsustainable costs (e.g., transport and lunch allowances). Instead, a peer network of

immunization HCWs, district officers, and regional-/provincial-level immunization leaders formed

through WhatsApp in both Arusha

Region, Tanzania, and Zambia’s

Southern Province. In Tanzania, we

observed HCWs using the

WhatsApp network to reach out to

peers about clarification on service

delivery indicators. The regional

immunization and vaccination

officer also used the group, to share

updates on immunizations, such as

the upcoming change from the oral

polio vaccine to injectable polio

vaccine. In Zambia, we found a

hybrid approach: HCWs first

connected in person at centralized,

classroom-style training session.

Participants shared their experiences, lessons, and feedback to help improve their

implementation strategies. HCWs then felt more confident to reach out to peers over WhatsApp

after having met in person. Within this virtual network, health staff formed their own WhatsApp

groups to enhance internal communication, such as a facility in-charges group. Two challenges

to this solution arose: In Tanzania, not all HCWs had smartphones or could engage face-to-

face. And in Zambia, while the number of HCWs connected through the virtual network

increased after installation of WhatsApp on tablets used by facilities for immunization service

delivery, some HCWs used up the data bundles earmarked for synchronizing to the ZEIR by

downloading social items such as photo, videos, and unnecessary applications.

Photo: PATH/Trevor Snapp


4

Recommendations:

If possible, hold at least one central training at which HCWs from the same district can meet

each other in person. This face-to-face sharing among peers can encourage ongoing virtual

communication given the rapport built during the central training.

Alternatively, use existing platforms, such as quarterly supervision visits, as opportunities for

cross-facility learning, as the supervision teams include facility-level HCWs. The district

could be deliberate about placing HCWs in teams to visit facilities where they can share and

learn.

In Tanzania, install WhatsApp on tablets with the EIR to provide access to the peer network

for HCWs without smartphones. To ensure data bundles allocated for exchanging

immunization data with the server are not depleted, a tool such as AppLock should be

installed to prevent downloading of unnecessary applications such as social media apps. In

addition, WhatsApp settings should be adjusted to prevent automatic downloads of photos

and videos.

Microtraining videos (Tanzania and Zambia)

Background: Four training sessions were provided to facilities in both Tanzania and Zambia to

introduce the data quality and use interventions and collaborate with HCWs on using data for

decision-making. As a way of providing ongoing training between those four sessions,

microtraining videos lasting two to three minutes each were developed to capture system use

and best clinical practices, as well as examples of data use. The intention was for the videos to

be filmed by HCWs as a way to demonstrate these topics to their peers and show the

improvements in service delivery performance that can result from the activities and lessons

learned. In Tanzania, TED Talks focusing on data use were downloaded onto tablets as a form

of educational entertainment.

Solutions adopted: The microtraining videos were deployed in both Arusha Region in Tanzania

and Southern Province in Zambia. However, they were not deployed as part of the package of

data quality and use interventions beyond Arusha (see the lessons learned and challenges,

below). The microtraining videos used in Arusha Region and Southern Province covered the

following topics:

Use of the EIR.

Clinical best practices, such as where on the child to inject specific doses of a vaccine,

how to weigh a child correctly, and how to administer immunizations.

Introduction to data visualization with images of how we see patterns in everyday

objects.

How to look for trends in data from the register and act on findings.


5

Lessons learned and challenges: As mentioned above, the goal of the microtraining videos was

to provide supplemental training for HCWs between the four onsite training visits from the BID

Initiative team. For example, in Tanzania, at the start of the rollout, the timing between the first

and second facility visit was three weeks. However, HCWs reported that they often did not

watch the videos due to an already heavy clinic schedule. Then, when the timing between the

first and second facility visit shifted from three weeks to three days due to changes in the

strategy for back-entry of data (see Back-entry of data), it became unlikely HCWs would view

any of the videos within such a short time frame. Another challenge was how to measure the

practical effectiveness of the microtraining videos, especially videos focused on the use of the

TImR and clinical best practices, as HCWs were unable to access the videos during an

immunization session to perform some of the activities explained. In Zambia, the microtraining

videos were not loaded onto tablets but shared during the second facility training, which was a

centralized, classroom-style training with HCWs and district staff present. The feedback

received was positive, since HCWs could see their peers as champions of the ZEIR.

Recommendations:

Create short (one- to two-minute) microtraining videos to increase the likelihood of HCWs

watching them. When providing reference materials, consider the method of delivery and the

accessibility of that method. For instance, information may be available on work hardware,

such as the tablets provided during the BID Initiative’s launch; however, this may make the

information difficult to access outside of normal work hours, when HCWs may be more

available to refresh their understanding.

Make time for HCWs to watch microtraining videos during centralized, classroom-style

training sessions.

Plan for the implementation team to help develop the videos, since it is challenging for

HCWs to feel comfortable and confident in front of the camera.

Take advantage of already existing forums, such as in-charges meetings, data review

meetings, and district quarterly review meetings, to provide a way to support HCWs in the

areas covered in the videos (e.g., use of the EIR and best clinical practices).

Data visualization, dashboards, and reports (Tanzania and Zambia)

Background: Data visualization helps to condense large amounts of data into non-intimidating

components that are quickly and easily understood. HCWs can more easily identify trends and

patterns in larger data sets, identify story narratives that explain the data patterns, and plan for

potential solutions to the observed challenges. While HCWs are already familiar with data

visualization through immunization monitoring charts, the focus of this intervention was to help

HCWs expand their understanding and use of large amounts of data presented through

dashboards (both digital and paper). The traditional format of the monthly aggregate paper

reports in both countries only allows a facility to see the data for one month, in numeric form,


6

without visualizations that allow for data analysis or insight into the meaning of that data. HCWs

were unable to create this sort of visualization themselves because they did not always have

access to data for the previous month and sometimes did not have paper to use to create

graphs for their own visualization.

Solutions adopted: The functional

requirements in both Tanzania’s

and Zambia’s EIRs include

dashboards for key indicators such

as dropout rates, stock on hand,

and coverage. These dashboards

are accessible both on the tablet for

facilities to review their service

delivery performance against

targets as well as on the web

application at the district for district

leaders to assess the district’s

overall performance.

Lessons learned and challenges: In both

countries, before the EIR was deployed, we learned that by creating a simple graph with data

from aggregate reports covering several months, HCWs would begin to identify trends (e.g.,

more defaulters were reported in June than in July and August). We suggested using a calendar

or even the back of an old poster to plot aggregate data over several months to see trends, if

HCWs had access to the data. This was a creative way to use the data immediately, rather than

waiting for new dashboards from the EIR. As HCWs better understood the process of

interpreting their data, we encouraged documenting the data on available paper to at least

capture and use that information. In the EIR, the use of stoplight colors to indicate performance

on key indicators is an intuitive way for HCWs to understand their data. For instance, in

Tanzania, as HCWs began to understand the colors in relation to the data and service delivery

performance, they started to ask questions about that data, such as: If we vaccinated so many

children this month, why is our coverage in yellow? Sharing and discussing reports and data

trends from the TImR generated at the facility level with HCWs helped to improve on data-based

action-taking as well. For example, the defaulter report from the TImR has helped HCWs to

more easily access and trace children, as the report contains enough information to trace.

Recommendations:

Be creative and meet HCWs where they are in their capacity for data analysis and how to

use data dashboards for decision-making. Listen to HCWs as they identify existing

challenges in using and documenting data (e.g., no graph paper), and work together to find

creative solutions (e.g., recycling paper and old posters no longer in use at the facility). The

Photo: PATH/Chimwasu Njapawu


7

importance or significance of data and its usefulness becomes apparent as HCWs walk

through the steps of processing and understanding the data. Therefore, a focus on the

process of using data, such as knowing what data to use, identifying trends, and discussing

possible solutions based on what is seen in that data, is useful when encouraging data use.

When creating dashboards, use stoplight colors to identify areas needing attention or action.

This helps HCWs to quickly assess data and allows better understanding and engagement

with the data, prompting staff to ask questions about the data. The use of stoplight colors

(e.g., red, yellow, green) on performance-based reports in an EIR is intuitive to health care

staff at facilities in Tanzania and Zambia.

Data use video games (Tanzania and Zambia)

Background: The team looked for free analytical tools in the form of video games that could be

accessed by HCWs using the tablets, in order to improve decision-making capacity. One key

game found was “Hospital Hustle,” in which a nurse has to triage patients as they come into the

hospital. The goal was to employ a fun way for HCWs to understand how data are used all the

time to make important decisions such as triaging patients, and how that same skill set can be

used with current facility data to improve service delivery.

Solutions adopted: “Hospital Hustle” was used during the test period but was not included as

part of the suite of interventions launched by the BID Initiative team.

Lessons learned and challenges: One challenge with the video games found online was the

requirement to pay for licenses, the cost of which would not be sustainable on health facility

budgets. In Tanzania, the games were not used due to the Internet/data requirements, as well

as the cost of the license fee. The user advisory group(s) also flagged concerns that the games

could become a distraction for HCWs, and negatively impact service delivery. Games were also

seen as depleting tablet batteries quickly.

Recommendations:

Consider video games with an analytical focus, with the awareness that cost may be

unsustainable unless a free game is found.

Encourage the use of non-traditional learning methods to engage HCWs. However, it is

important to work with the user advisory group(s) to understand some of the unintended

consequences of having games on tablets, such as distracting HCWs from attending to

clients.


8

District data use guide (Tanzania and Zambia)

Background: One of the biggest challenges to building data use is the disparity in capacity from

HCW to HCW, facility to facility, district to district, and up to the national level. Variables such as

staff turnover, individual capacity, and limited training factor into these challenges. Data use

guides were proposed as succinct job aids to provide easy guidance on the data elements

generated at the facility level. These guides would assist users with the interpretation of data

and interconnectedness of the data elements, and how they can be used in their daily work.

Solutions adopted: Guides were developed for both the facility and district levels, with a focus

on the data elements used at each level. The facility-level guide focused on which data were

currently being used to make decisions and how often, as well as the impacts of those

decisions. The district-level guide focused on which key areas to target when reviewing facility

reports as well as hypotheses about what HCWs may be seeing in the data trends. Both

Tanzania and Zambia deployed these guides and they are currently in use.

Lessons learned and challenges: In Zambia, the guide became the champion document

because the information was relatable to facility-level data, and the benefits of the guide were

seen and felt immediately. It was very important, however, to be clear with HCWs that the guide

was not meant to be part of an assessment but a way to learn and share lessons from the

process with their colleagues. The guide was planned to be introduced at the second “touch”

(i.e., facility visit) of four touches per facility, but ended up being used as early as the first touch.

While going through the guide, HCWs began to understand the normal processes of collecting

and using data in service delivery. Sometimes the HCWs would check standard operating

procedures to be sure. HCWs began healthy debates about the process itself; and in those

conversations, would pool their knowledge to come up with a shared solution. What our team

observed in this process was that HCWs had not received training for some time, so this was an

opportunity to provide a refresher on best practices and data use.

Recommendations:

When working with HCWs and district staff on the data use guide, be sure to clarify that it is

not an assessment tool. When HCWs feel they are being assessed, it limits transparency

and candidness, and hinders the learning process.

Be sure to use actual facility data when going through the guide with staff; this creates a

greater feeling of ownership.

Ensure that the demand for data from the higher level is strong, as this drives data use

across the health system.

Having a connection to any available electronic tool (such as the EIR) and any specific

decision support available in this tool is important. Institutionalization of the data use guide

as part of the reporting process and supportive supervision is vital for data accountability

and sustainability of the intervention.


9

Data use posters (Tanzania and Zambia)

Background: Motivating HCWs to see data use as part of their responsibilities and as a way to

improve immunization service delivery was part of the BID Initiative’s work to create a culture of

data use. Within this, one of the interventions tested by the teams was data use campaign

posters. These posters were meant to motivate HCWs with specific phrases during each facility

touch as a way to match the progression of data use across the four touches.

Solutions adopted: In Tanzania, three posters were developed and printed in Swahili.

Data awareness: Notice regarding changes to improve data quality and use across the

health system.

Data quality and data use: How data quality and use are important for improving and

providing services.

Barcode awareness: This was created during implementation as a way to address

community fears about why barcodes were being used.

The following posters were used in Zambia.

Better services: A focus on good data leading to good decisions and better planning for

health services.

Data as an eye opener: Included a photo of an HCW reviewing data.

Work not documented is work not done: Promoting the documentation of services provided.

Lessons learned and challenges: In both countries, we learned that the posters were not

effective in driving a culture of data use. HCWs were very busy during service delivery and did

not always read the posters. In many facilities, the posters were lost among other posted

materials.

Recommendations:

Use vocal ways of conveying a data use campaign, such as radio or messaging points to

HCWs.

Ensure change management practices are in place to create a culture of data use, such as

supportive supervision and a demand for data from higher levels of the health system.

Workflow redesign (Tanzania and Zambia)

Background: Workflow redesign was intended to address some of the bottlenecks in

immunization processes at clinics. The average wait time for caregivers whose babies did not

require immunization decreased because facility staff were able to better manage patients. In a


10

busy urban setting, workflow redesign was especially important, since caregivers were more

likely to receive care at multiple immunization tables in a day over the course of the baby’s visit

to the facility. The burden on HCWs to separately capture and collate data for aggregated

reporting was high initially and could be addressed through workflow redesign.

Solutions adopted: Brainstorming strategies were applied to come up with suitable workflows for

any particular facility, with emphasis on where the tablet would be placed and how to improve

workflow efficiency to benefit both patients and HCWs.

Lessons learned and challenges: Standardized operating workflows were initially proposed

according to the facility category and whether the facility served a high or low patient volume.

However, practical implementation on the ground proved different, as individual facilities had

their own challenges based on environment. For example, at Libuyu clinic, the proposed

workflow was not conducive because of the infrastructure of the building and staffing levels.

Therefore, workflow redesign should be done onsite in consultation with facility staff. When

health staff appreciate the concept of workflow redesign, they are able to refine and apply the

same in any situation (e.g., static or outreach stations) to make their work easier. Districts

appreciated the concept of redesigning workflow to suit each unique environment, especially in

clinics with limited staff.

Recommendations: Even when modifications/improvements in workflow redesign are obvious,

do not propose solutions to HCWs immediately. Instead, provide a platform (e.g., through

brainstorming and asking direct questions) so that HCWs identify their own challenges and

solutions for a better work flow, for enhanced ownership and sustainability.

Data use best practices (Tanzania and Zambia)

Background: One component of the BID Initiative intervention package was a data use

campaign aimed at strengthening adoption of data to inform decision-making. Multiple tools and

activities were tested to determine the best method for encouraging data use.

Solutions adopted: We tested several interventions and deployed a subset to motivate HCWs to

use data. The final set of interventions was then used in targeted data use campaigns

throughout launch, including such interventions as peer networking, data use guides, targeted

supervision, workflow redesign, and data visualization.

Lessons learned and challenges: Only certain aspects of the data use campaign proved to be

relatively effective and were retained for larger deployment. Several lessons were learned

around the data use work when introducing interventions at facilities.


11

Use the current data available in the facility or district when training on data use. By using

data already in the paper registers at facilities, the training provided an opportunity to tease

out and focus on current challenges as an initial step to build data quality for what is then

put into the EIR. This also helps in identifying challenges in data use that can be shared with

the district as part of supportive supervision, so supervision is targeted to the specific needs

of HCWs.

Normalize data visualization. This includes

framing data use from the perspective that

we can see patterns in things (i.e., identify

trends) and use data (e.g., to manage talk

time), and this can translate into aspects of

data use for service delivery. We also

framed that data use is an ingrained part of

health care rather than an added step to the

work.

District immunization and vaccine officers

have managed to use their data from EIRs

by sharing examples of data use in best-

performing facilities with HCWs. This has

generated discussions regarding the data

trends of underperforming facilities and how

to improve.

Sometimes errors in data come down to

misunderstanding of arithmetic. Although

EIRs perform calculations, strengthening this

skill across the health system is important,

especially when HCWs rotate between

clinical care services and need to calculate

different indicators for service delivery

performance.

Calculations done by the TImR and ZEIR

have freed up HCW time to visualize their data, compare trends, and more deeply discuss

data quality improvement. Doing calculations by hand was time consuming and left little time

to go through the data results and implications.

Recommendations:

Be patient when working with HCWs. Everyone is at a different level of using and

understanding data. Providing multiple interventions reinforcing the same skill are important,

as people learn differently, and mastery grows with repetition.

Photo: Bill & Melinda Gates Foundation/John Healey


12

Support HCWs with an improved EIR that generates actionable data to help them to use

their data easily. For example, defaulter reports and alerts regarding nutritional status and

stockouts from EIRs has allowed HCWs to take action more efficiently.

Change Management

Change readiness assessment (Tanzania and Zambia)

Background: The ability and openness to change can vary from individual to individual and

fluctuate over time. The ability to understand the current context can have a strong effect on

program adoption for those being asked to change. The change readiness assessment

measures current resources and gaps within a specific context that would enable or hinder

change. A high score on change readiness increases the likelihood of the change succeeding.

The Change Readiness Assessment Tool was first developed in Zambia and later used in both

Zambia and Tanzania.

Solutions adopted: Two versions of the tool were developed and used for the change readiness

assessment. The first version consisted of questions with responses scored on a Likert Scale.

The score was used as a way to determine an HCW’s readiness for or reluctance to change and

informed the implementation team of how to engage with that HCW. A second version of the

readiness assessment came later and was designed with open-ended questions to elicit

conversations about factors to succeed in implementing new ideas at the facility, challenges

with implementing those ideas, and the resources needed for those changes. The outcomes of

these assessments were mapped to Kotter’s 8-Step Process for Leading Change and used by

the implementation team to understand how to continue to motivate HCWs along the process

steps.

Lessons learned and challenges: Answers to the initial questionnaire, scored using the Likert

Scale, led to incorrect assumptions about HCWs’ readiness or not to change. The second

version included open-ended questions, such as: What factors have helped you successfully

implement new ideas at this facility? What resources (people, finances, tools) do you have

access to that can help you to overcome challenges when implementing new ideas? These

open-ended questions facilitated feedback and sharing of challenges for HCWs and the BID

team to address together: for instance, stockout of stationery for a month, but data can be

documented anywhere. If displayed prominently, the data will be seen and used. The open-

ended questions also provided a platform for discussions with HCWs and became a nice way to

introduce activities for each touch. The tool became more useful because HCWs could share

and discuss real rather than perceived challenges. Within the Tanzania context, a unique

approach was adopted to assess readiness to change, rather than administering the change

readiness tool. The introduction of interventions was generically designed to address intrinsic

https://pathseattle.sharepoint.com/sites/BID/Pics/BID%20Briefs%20-%20Graphics/VAD_BID_LL_Change-Fig1.jpg?csf=1&e=qKtYnY


13

and extrinsic value of data quality and use regardless of HCWs’ positions in the readiness to

change matrix. This provided room for HCW self-reflection in relation to data quality and use in

their work and the team capitalized on this as a foundation to improve their practices on data

quality and use interventions.

Recommendations:

Use open-ended questions to fully grasp the spectrum of responses to enable better

understanding. Closed-ended questions result in limited responses. Through inquiry and

understanding, program staff are able to gain trust and involvement with facility staff. This

allows facility staff to lead discussions and speak from their experience rather than a partner

team coming in with assumptions.

Be open and creative when coming up with solutions with HCWs.

Community understanding of barcodes (Tanzania and Zambia)

Background: With the introduction of new tools in facilities, such as tablets, barcodes, and quick

response (QR) codes, we learned that it was important for community members to understand

what the tools were being used for and what this meant for the services they receive. We

developed messaging on community sensitization for HCWs to explain the tools.

Solutions adopted: In Tanzania, we developed messaging about the barcodes to explain to

communities why they were being used (e.g., to ensure children receive lifesaving vaccines). In

Zambia, we also developed messaging about the QR codes, relatable to communities by

comparing QR codes with the barcodes used at the grocery store to make for a more efficient

checkout process.

Lessons learned and challenges: Where community sensitization was most helpful was around

the barcodes in Tanzania. Some communities perceived the barcodes to be a way to mark their

children and became fearful of bringing their children in for immunization services. However, by

having the messaging about the barcodes come from HCWs, trusted by the communities, these

communities felt more at ease about the use of barcoding. The messaging included examples of

how barcodes ensured that children received the correct dose and could create efficiencies in

service delivery by ideally reducing waiting times for caregivers. In Zambia, caregivers were not

too concerned about the QR codes, but sensitization was helpful in encouraging caregivers to

permit HCWs to take their child’s photo for the system. Some caregivers declined, and other

caregivers were okay with the photo being in the system. What the team learned is that it is

important for the HCW, someone known in the community, to explain to caregivers the benefits

and reasons behind QR codes, barcodes, and photos. With increased use of WhatsApp across

Zambia, the culture of digital tool use has become more acceptable, which could be why many


14

caregivers in that country did not

mind having their child’s photo

taken for the ZEIR. Partners did not

see the value of including

children’s photos in the system and

requested that functionality be

removed; however, given

caregivers’ feedback, this function

was retained.

Recommendations: As HCWs

come to own the process and tools,

work with health facility staff on

messaging for HCWs to convey to

the communities they serve

regarding why the new tools are being used. As countries choose paths to digitally transform

their health systems, they need to consider the health care consumers, who need to be made

aware of the importance of the new tools.

Parallel reporting systems can inhibit adoption of new tools (Tanzania and

Zambia)

Background: When introducing the new data collection tools (tablets) for the TImR in Tanzania

and the ZEIR in Zambia, the legacy paper tools were used in parallel with the EIRs as they went

through version updates and bug fixes based on user feedback and lessons from the

implementation. Months and even up to a year after implementation, parallel systems are still in

use. The challenge becomes more complex if the parallel system also includes another

electronic tool collecting similar but not identical data, as adoption is hindered not only by the

multiplicity of tools themselves but by external factors that might send confusing messages

around which tools to focus on and use. During a visit to Chile, BID Learning Network (BLN)

participants learned that the country’s process to remove the paper tools happened over years,

with the requirement that facilities consistently use the EIR 90 percent of the time. Chile

stakeholders recommended that rather than waiting for the entire country to reach this

performance level, as soon as a region reaches the target, the legacy paper process should

end. This was done in Chile to prevent early facilities from running parallel systems for years,

while facilities more recent to use the electronic process only had to use parallel systems for

several months. Not all countries have to follow this methodology, but this is one example of an

approach that can be taken.

Photo: PATH/Chimwasu Njapawu


15

Solutions adopted: Team members worked closely with the governments of Tanzania and

Zambia to address the use of parallel systems early on in our collaboration. Part of that process

included assurance that the EIRs could produce the data needed for the patient-level paper

tools at the facility as well as the aggregate reporting tools used from the district to the national

level. Both registers were also built to exchange data with District Health Information System 2,

the national-level reporting tool in both countries. We are seeing progress in Tanzania as the

country scales. The government identified key indicators to measure progress as well as what

targets facilities must meet to discontinue use of legacy paper tools. As of March 2018, the

government of Tanzania had determined that 33 facilities across Tanga Region could

discontinue use of a set of identified, legacy paper tools for immunizations. Zambia is beginning

a similar process to determine performance indicators to remove paper tools when facilities

meet certain targets.

Lessons learned and challenges: The challenge of not setting expectations is that both systems

(legacy paper process and the new digital process) are used infrequently and data quality

overall can suffer as a result. In addition, HCWs may lose motivation to use the new tool

because running both systems in parallel presents a greater burden.

Recommendations:

Set clear expectations and guidelines and gain agreement at the ministry level prior to

implementation and communicate to HCWs so they know what is needed to discontinue use

of parallel systems.

Develop a clear roadmap with health ministries during the transition to paperless, with

milestones to both aim for and document progress. This can include system and logistical

changes, but also can be paired with areas such as policy support and standardization.

Communicate clearly to HCWs that the parallel process will be burdensome in the short term

but the roadmap outlines steps to full transition to use of only the digital system.

Identify stakeholders who will own and lead this process to better ensure accountability for

this shift.

Rollout/Implementation Strategy

Touches strategy: Timing of facility (onsite) touches or classroom-style

touches (Tanzania and Zambia)

Background: The BID Initiative was built following an implementation model that consisted of an

on-the-job training approach organized into sets of touches (visits), each with a predetermined

set of activities at both the health facility and district levels. The timing of the facility and district


16

touches was strategically set to ensure service provision was not compromised. These touches

provided a phased approach to learning of interventions; encouraged district ownership,

leadership, and involvement; and encouraged the progression of data quality and use and

decision-making for facilities (see Figure 1).

Solutions adopted: Tanzania began rollout in Arusha Region with four facility touches led by BID

Initiative staff. The last three districts in Arusha Region and additional regions (Kilimanjaro and

Tanga) were led by district government staff data use mentors (see Data use mentors below).

Zambia also began with four touches to facilities and went through three phases of changes

during launch.

Lessons learned and challenges: Tanzania and Zambia began with four facility touches and

implementations led by BID staff. As lessons were learned, the methodology was adapted. In

Tanzania, in the final districts of Arusha Region as well as Tanga and Kilimanjaro Regions,

implementation was led by district-level government staff trained by the BID team to continue

the rollout process (see Data use mentors). This strengthened government buy-in and

ownership. In Zambia, implementation occurred over three stages. The first stage began as the

ZEIR was being developed. The change management strategy to improve the data use culture

was launched in the first three districts of Southern Province. This created a focus on tools such

as data visualization, clinical workflow redesign, and the data use guide for reporting. The

second stage occurred when the ZEIR was ready for use in facilities. Rather than having all four

touches be onsite at the facilities, the second touch in this hybrid model occurred in a

classroom-style training that brought together representatives from all facilities within the same

district as well as district leadership to focus on using the ZEIR and the data in the system. The

third stage began in late 2017 with a focus on scaling the ZEIR to all remaining districts in

Southern Province. All remaining facilities received their initial touches at this stage, bringing

use of the ZEIR to all 254 facilities providing immunizations in Southern Province. The final

touches were then launched in early 2018 with a strong focus on supporting HCWs in use of the

data in the ZEIR, including for decision-making.

Recommendations (refer to the BID Initiative brief on the rollout strategy):

Starting with the district engagement and support helped ease the implementation process

and allowed the district to conduct supportive supervision in facility sites simultaneously,

enabling sustainability. The deployment of the touch strategy is dependent on

circumstances on the ground. If there is significant commitment from district-trained mentors

and district staffing levels are sufficient, onsite training is ideal to reach out to as many staff

as possible at a facility while allowing continuity of service delivery at the point of care.

Engaging district staff during implementation is both effective as an entry point to facilities

and increases sustainability. When the district was onboard with the activities, HCWs were

http://bidinitiative.org/wp-content/uploads/VAD_BID_LessonsLearned_Rollout_v1_rev04.pdf


17

more likely to make time to work with BID staff during the touches, which were part of onsite

training.

Involving all health personnel found at the facility while conducting onsite training allows the

maximum number of people to be trained and increases efficacy of information-sharing.

The figure below depicts the timeline of the original plan for touches at the district and facility

levels in Tanzania and Zambia and reflects their consolidated strategies; each country adapted

this timeline based on the country-specific needs discussed above.

Define what makes a touch successful (Tanzania and Zambia)

Background: Each facility was budgeted to receive four touches during the rollout process. As

the BID Initiative team began deployment of the data quality and use interventions, it became

evident that clearly defining the activities and expected outcomes of each touch would ensure

consistent delivery of the activities for each touch as well as the overall success of the touches

both independently and collectively.

Graphic: PATH/Shawn Kavon


18

Solutions adopted: To create a consistent method of delivery and assessment for each touch, a

checklist was developed for implementers to reference and complete during each touch.

Checklists included the necessary topics each implementing staff member should cover during

a specific touch. Monitoring and evaluation forms were also developed and utilized for

measuring success.

Lessons learned and challenges: Conducting feedback sessions after touches and sharing

experiences helped the BID team to redesign the strategy for effectiveness and continuous

improvement. This was especially true for the first districts of the implementation, where

feedback sessions occurred daily. As the team discussed what worked well, what did not work

well, and what could be improved, the touches became more effective.

Recommendations:

Develop a set of talking points for all team members to cover during each touch to ensure

each facility receives the same attention, information, and quality across the implementation.

All team members should cover this important content, but it can be delivered in their own

style and for the context of the specific facility or district.

Create a checklist of what activities and tools are needed for each touch. Given HCWs’ busy

schedules and the onsite training approach, sometimes a few areas may not be covered.

Using the checklist, teams can identify which activities were not completed or which tools

not shared so that during the next touch the team can focus on ensuring those aspects are

covered.

Define success indicators. By creating a tool to measure the success of each touch, the

team was able to use that data to modify parts of the rollout and/or touches through an

iterative and adaptive process. Tools thoughtfully designed for implementation allow staff to

use the data from those tools to facilitate a process of adaptive management. Role-playing

was a useful tool in building staff confidence and capacity to deliver touches. The rehearsal

of these trainings ensured that core messages, such as objectives, were communicated in a

consistent way.

Data use mentors (Tanzania)

Background: Capacity development is a key element of facilitating improved access to

sustainable and high-quality immunization services. The BID Initiative began with an on-the-job

training approach (see Touches strategy); PATH staff visited facilities to provide training to

HCWs. The training across the four touches was based on a training plan prepared with support

from Council Health Management Teams (CHMTs), which serve as district supervisory bodies.

The implementation team conducted facility touches accompanied by CHMTs in the first five

districts of Arusha Region. Throughout the launch, the team met to review progress and identify

areas for improvement. It was during these reviews that a new approach was identified to


19

increase rollout speed, ensure sustainability, and reduce costs, resulting in a more scalable

model. This model was initially used in the last two districts of Arusha Region and later in Tanga

and Kilimanjaro Regions.

Solutions adopted: Whereas the initial touches in the first five districts were completed by BID

staff, following the new model used in the last two districts of Arusha Region and beyond,

touches were completed by district-level government staff called district data use mentors

(DDUMs). This was done to increase government ownership and encourage sustainability

beyond the BID Initiative. This was a viable solution under Tanzania’s decentralized

government; sufficient human resources were available at the district level due to the greater

capacity for districts to be decision-makers. The government individuals selected for this role

were identified by district health management teams as knowledgeable regarding the health

sector, data, and basic computer functions. As part of this shift, the BID team employed a

training-of-trainers model, partnering with and training DDUMs to conduct facility touches with

the suite of data quality and use interventions. DDUMs received training and certification of

training completion, and thereafter were partnered with two BID staff for ongoing support.

Ultimately, the initial five districts in Arusha Region also identified DDUMs, who were then

trained by BID so that the entire region followed this model. Today, DDUMs provide ongoing

supportive supervision to facilities, especially in system use and adoption, as well as ongoing

technical support for data use.


Originally, CHMTs were to provide ongoing support to HCWs during implementation.

However, due to the number of facilities and workload, CHMT members primarily focused on

their respective technical areas and did not have time to provide the needed support to

HCWs in areas outside their focus. Therefore, CHMTs took on a supervisory role to ensure

effective use of the EIRs and related interventions, as opposed to providing regular and

timely technical support to health facilities.

In the original strategy, the BID implementation team worked with one district at a time

through the first three touches. This approach took, on average, three to four months per

district. As a result, the BID team was unable to work in multiple districts concurrently. In

addition, due to team bandwidth limitations, they were unable to provide timely support to

struggling health facilities in districts they had already visited. CHMTs were expected to

address facility issues after facilities completed the third touch; however, CHMT members

were unavailable to provide this support.


20

Although two separate teams,

BID staff and district staff, were

responsible for assisting

struggling facilities, needs of

facilities still were not met. We

learned that we must ensure

government staff at the district

level are capable of providing

needed support to HCWs on

data quality and use

interventions at the facility level,

when required.

In the original strategy, the BID

implementation team trained

CHMTs to take an active part in launch and ongoing support. BID Initiative staff were also

responsible for developing the data use capacity of HCWs through frequent visits to

facilities, but without CHMT capacity to provide ongoing support, it was difficult to ensure

smooth transitioning of the BID implementation roles from PATH to district authorities to

maintain and sustain the initiative. In the new strategy, DDUMs took an active role in the

training of HCWs as well as providing ongoing support to health facilities. This allowed for a

smooth implementation capacity transition from BID Initiative staff to the district through both

training and practice as part of the rollout process.

In the original strategy, increasing the rollout pace meant increasing the size of the BID

implementation team to cover multiple districts at the same time or reducing the number of

weeks they spent in a district. Developing district capacity to conduct the rollout using

DDUMs increased the pace of the launch while maintaining a manageable BID team size.

Recommendations:

Make time to review and adjust implementation strategies (e.g., what works; what could be

improved). Lessons from the original rollout strategy came out of a learning and appraisal

meeting led by key staff from the health system in Arusha Region. This provided a depth of

understanding; the decision to shift to using DDUMs came from the lessons and

recommendations shared in this meeting. When possible, convene key persons for

feedback and input; this can alter the course of activities in a positive way.

When considering staff in the health system who can lead the implementation, maintain a

realistic understanding of work demands. CHMTs were too busy to provide ongoing

technical support to HCWs but could manage to act in a supervisory capacity during

implementation, while district staff who were already following up with facilities more

naturally fit into the role of DDUM. A mix of CHMTs and non-CHMTs in the DDUM-type role

Photo: Bill & Melinda Gates Foundation/John Healey


21

is the preferred combination, as that will allow for both smooth implementation and

supervision.

Determine the human resources capacity within the health system and if there is sufficient

capacity to implement a strategy like this. As an example, Zambia does not have the same

decentralized model as Tanzania, and therefore districts do not have sufficient staff to

implement the DDUM strategy. Adaptations were made with the group touch with facilities

instead, and district-level involvement (but not sole leadership) in the touches. Adapt to the

human resources available at the district level to lead and implement this type of strategy.

Training BID and district-level Ministry of Health staff to understand the

interventions through role-play exercises (Tanzania and Zambia)

Background: A classroom-style approach was originally selected when BID Initiative staff went

through training to deliver the data quality and use interventions to facilities. This training would

cover each intervention: the purpose, the tools, the outcomes. After the initial week of

classroom-style training, the team added more time for role-play exercises. One team member

pretended to be a HCW as the other team members acted as BID staff providing touches. The

team became familiar with and created strategies to address HCWs who would be resistant to

change and those excited about change, and how and when to interact based on busy clinic

days. Overall, role-playing introduced the need to be flexible in the way interventions were

presented during touches based on the facility environment.

Solutions adopted: This type of role-playing was incorporated into all training for both BID and

district-level staff involved in the implementation (DDUMs and others).

Lessons learned and challenges: The implementation of this technique helped prepare the team

for facility visits and for the unconventional instructional setting. This training also helped to

identify strengths and weaknesses in the staff’s training style and allowed a safe space to

resolve the weaknesses. Lastly, the exercise instilled confidence among team members in their

abilities before they visited the health facilities.

Recommendations:

Focus classroom-style trainings on uniformly instructing all staff on the purpose, tools, and

intended outcomes of the interventions. This will ensure everyone has the same

foundational knowledge of each intervention and how the interventions will evolve with each

touch and complement each other.

Allow two days for role-play exercises to ensure adequate time for both practice and

feedback to team members.


22

Peer learning network

Embedded within the BID Initiative is the BLN, an Africa-led and member-owned peer-to-peer

learning network designed to share lessons learned and assist in the development of common

tools, practices, and policies to enable scalable data solutions. The BLN Communication

Strategy outlines several channels of communication through which peers can share information

and knowledge. These include face-to-face meetings and virtual platforms.

Have a community manager in place

Background: The BID Initiative hired a manager to ensure the effective management and

coordination of the BLN for participation and engagement of members from across the globe.

Originally, the BLN engaged a consultant who rendered intermittent and inconsistent support,

not conducive for the building of a strong community.

Solutions adopted: The BID Initiative recruited a full-time community manager to enable

effective management and coordination of the BLN community.

Lessons learned and challenges: Managing a virtual and peer learning community comes with

challenges such as geographic, cultural, and language differences. A dedicated coordinator is

essential for building collaboration, learning, and information-sharing spaces as well as trust-

building among interacting members through virtual and in-person meetings. Peer learning is a

continuous and iterative process with no prescribed path and requires adjustment and

refinements as the needs arise. A challenge that arose was that as the network grew, the part-

time consultant managing the community was not able to monitor the growth and changing

needs of the network. Consequently, member needs were not identified and addressed in a

timely manner. Surveys that were critically essential to determine progress and understand

member needs and preferences were not regularly conducted. This limited the learning process

and valuable lessons were likely lost.

Recommendations:

To create a robust virtual community, closely monitor and engage the community to ensure

processes are running smoothly and to the standards of the community. Ideally, the

structure for engagement should be paired with a monitoring and evaluation plan to track

progress and strengthen strategies to improve the value of the peer network. A coordinator

who can focus on and manage these processes is vital to shaping this structure and triaging

feedback from members. The coordinator needs to closely monitor the evolution of the

network over time to match the needs of the community.


23

Conduct periodic surveys to gather information as a way to understand the benefits of the

network and what participants want to learn. The results of these surveys can be used to

enrich the network’s own learning process. Continued contribution from members is

essential to a thriving community.

Webinars

Background: Webinars on different technical topics were held monthly with different countries to

share their experiences of success and failure and lessons learned on their projects and

programs. Webinars were first conducted by BID Initiative staff, but later were BLN country led.

Solutions adopted: Webinars were held once a month; on special request, two webinars were

held within a month. An annual webinar schedule was established to maintain the cadence and

to coordinate appropriate presenters. Presentation guidelines were also established, to promote

successful webinars. Reminders were sent over Google groups and WhatsApp to inform

members of upcoming webinars.

Lessons learned and challenges: Good publicity, constant reminders, and timely invitations

helped to increase the number of webinar participants. Presenter confidence, resulting in

higher-quality presentations, was boosted through practice runs prior to delivery of the

webinars. The use of multiple platforms for communication and webinar reminders allowed

everyone to maintain communication within their comfort level. Webinar invitations were sent

through Google groups; however, this was not a seamless solution. Most BLN members resided

in countries with limited internet connectivity, making it difficult to access Google group

invitations and alerts. Language remained a barrier in group communication among the

Francophone and Anglophone countries, which was a challenge to moderate. Thus far, the

Francophones were presented in English, which created a somewhat unequal level of access to

BLN information.

The shift from BID-led to BLN country-led webinars increased the participation of country

members and each was eager to present on their experiences implementing digital health tools.

Participant sign-up for these presentations was robust, with a 12-month waitlist to present.

Recommendations:

Package information and lessons learned from the webinars as a valuable resource for

running a virtual community.

Provide written materials for easier translation to be discussed in the webinar, to help

promote greater engagement with the webinar and to ensure the non-English speakers have


24

an opportunity to share information over webinars in comfortable languages. These

materials can be shared through existing virtual platforms such as the BLN Google group.

Communication mediums and engaging the community

Background: We began and continue to focus on keeping discussions active on the BLN Google

group, with a membership of close to 450 people across the globe built over the past four and a

half years.

Solutions adopted: Due to challenges with access to the Internet to participate in the Google

group, BLN country members created a WhatsApp group at the discussion meeting in Zambia in

2017 to use in addition to the Google group, which remains as a larger list of global, regional,

and country partners.


While the Google group has been quite effective in reaching out to members from global

agencies, funding agencies, and nongovernmental organizations, it has not worked as well

among the BLN country members themselves. Many of the BLN members spend an

extensive amount of time working in the field, where Internet access can be limited and

unreliable. What we found is that WhatsApp is also an effective medium for our BLN

members. In an online survey conducted to learn what would be preferred by country

participants, we requested a recommendation for the most suitable communication medium.

Members overwhelmingly responded with WhatsApp as the highest-trending, most

affordable, and accessible platform within the African context. A majority of members have

access to a smartphone and are familiar with the WhatsApp platform through their social

and other business communication.

At the BLN discussion meeting in Lusaka in September 2017, BLN members established a

WhatsApp group and identified a bilingual member to moderate discussions, with oversight

from the BLN community coordinator. This solved the language disparity for Francophone

members who did not have access to the Google group comments, allowing them to

communicate in French. However, it has been a lot of work for the volunteer moderator, who

is assisted by the community coordinator. The coordinator and the moderator do exchange

notes when the moderator is available.

The WhatsApp platform is used to send out notifications and alerts for discussions,

webinars, surveys, and other announcements that are posted on the Google group, which

has created greater engagement in group discussion than when the schedule of topics was

disseminated only through the Google group. The group has engaged in discussions,

learning, and questions around immunization information systems and logistics/supply

chains over WhatsApp.


25

Responding to BLN country member needs and tailoring the communication style to make

their engagement less taxing has resulted in a higher level of participation. Although the use

of multiple platforms has increased the administrative burden of managing the group’s

communications, it was a necessary step in the effort to ensure effective communication

with members within the African context. The addition of WhatsApp proved effective in

garnering member responses in a timely manner by encouraging members to check their

emails and Google groups and register for webinars. Alerts sent to the WhatsApp group

have been useful in ensuring BLN members respond to surveys posted on other platforms.

Despite the advantages of using WhatsApp, some members are still not taking part in the

discussion boards. This could be due to the high level of work expected of BLN members

outside of BLN activities.

Recommendations:

When establishing a learning group, determine the comfort and accessibility the group has

with different communication platforms. Members were much more engaged after the

platform for communication shifted, proving that their disengagement was not due to

disinterest, but due to inaccessibility. The best solution for a group may be one in which the

group has multiple ways of posting and discussing on platforms, which allows cross-posting.

Understand the limitations of the communication platforms. There are approximately 180

BLN individual members, some of whom are currently part of the WhatsApp group, and

since maximum membership on any given WhatsApp group is 256, we may have to shift

platforms (or divide the group) if the group grows significantly.

Find out what interests individual and group members to create greater engagement.

Members have been requesting segmented thematic areas for engagement to allow for

engaging in relevant or interesting content or opting out of specific topic areas. This will

require volunteer members to manage subgroups and communicate with the community

coordinator.

Peer network governance structure

Background: Country ownership and leadership in the communal learning process were a

fundamental ethos of the BLN. It was critical that mutual learning occurred across BLN countries


26

to enable the design of appropriate

and acceptable solutions to data

quality, availability, and use

challenges that had been identified

by the community. It was important

in this instance to ensure a

governance structure that allowed

the country voice to be heard and

shared.

Solutions adopted: Initially, we

adopted a loose governance

structure that included a director, a

community manager, and a peer

advisory group composed of six

country representatives (three from West Africa and three from East/Southern Africa). In

addition, three-tiered country membership included the two demonstration countries (Tanzania

and Zambia) that pioneered the design and implementation of interventions to improve data

availability, quality, and use; 20 design countries that formed a collaborative and invested time

and intellectual resources in interventions that were being designed by the demonstration

countries; and discussion countries and members that accessed information coming out of the

BID Initiative and the BLN.

Lessons learned and challenges: The loose BLN governance structure sufficed for the initial

purpose for which it was set up. However, BLN members noted that the current BLN structure

has been inadequate to address issues beyond immunization, follow up on activities and

progress in design countries, and enable sustainability and scale of identified solutions. It was

further noted that the BLN had not galvanized optimal support from donors, regional bodies, and

African governments—a key component for sustainability and scale.

Recommendations: BLN members made the following recommendations in relation to creating a

stronger BLN governance structure:

The focus of the BLN was too narrow; the BLN could be utilized in an expanded role that

goes beyond immunization data alone.

There could be benefits in having a mechanism to better follow up activities in countries

beyond the initial demonstration countries; one potential way to do this would be to have

country focal persons reporting to the secretariat from member countries to augment inter-

country learning and accountability of countries to the network. These country focal persons

would promote and advocate for the BLN in-country, as well as collect and share key



27

information on in-country progress where solutions to improve data availability, quality, and

use were being implemented.

The BLN would benefit from a steering committee with representation from countries,

partners, and regional economic bodies to enhance sustainability, increase access to a wide

range of expertise, and enhance the ability to strengthen political will to adopt and scale

promising practices in BLN countries.

Ideally, countries could show greater commitment to the BLN by committing personnel and

by making monetary contributions to the activities of the network.

Sustainability and Scale

How Tanzania and Zambia are scaling

Background: From the earliest planning stage, each BID Initiative intervention was designed

and implemented with a focus on sustainability and scale. This is important because once a

program is started, shifting to account for sustainability and scale can be difficult and costly. The

BID Initiative focused on country representation and involvement from the beginning, with

specific government roles established to transition to full government ownership at the end of

the project grant.

Solutions adopted: Designing the interventions with a health system–thinking approach allows

for the consideration of the needs, capacity, and processes already embedded within the health

system. Interventions were designed from the perspectives of both frontline HCWs and

policymakers through the user advisory group(s). Design of EIRs considered openness and

existing in-country technology that would help to sustain the program beyond the project as well

as the capacity of end users of the system. Government ownership of the BID Initiative included

data use mentors trained by BID staff but employed through the government. This increased the

number of skilled staff within the government, ensuring the program would continue beyond BID

staff involvement. Additionally, close engagement with government from the beginning enabled

the BID Initiative to coordinate with governments on larger health system strategies. In

Tanzania, this resulted in inclusion of the interventions in Health System Strengthening grants,

as well as the Gavi, the Vaccine Alliance, Targeted Country Assistance grant. This support was

in a way to support additional scale in-country, and to proactively transfer BID Initiative technical

expertise to the government. In Zambia, the BID Initiative is coordinating with the government

on the Expanded Programme on Immunization to further support the EIR developed with BID

and the government and to seek funding for scale. Lastly, facilitating partnerships between local

technology partners and governments allows in-country technical support to improve the

stability of the program.


28

Lessons learned and challenges: Although the BID Initiative interventions were designed to

scale, the timeline of the interventions shifted drastically when appropriate technologies were

unavailable for implementation as originally planned. This delayed introduction and scale-up of

the interventions. However, although the intervention strategy shifted, the sustainability and

scale built into the program remained robust, with constant communication and coordination

with government throughout the process. BID Initiative staff also had transparent conversations

with ministries of health to understand their capacity to support the program beyond the BID

Initiative grant, including the costs, and how to best include ongoing funding in government

budgets or from other funding sources.

Recommendations:

Design interventions for sustainability and scale from the beginning planning stages. This

ensures that program funds are spent in the most effective manner to allow for full transition

to government ownership.

Engage the government as an active partner in the implementation strategy and a partner in

the design of the intervention to establish government ownership and promote government

adoption at the end of the grant.

Electronic Immunization Registries

Unique identification of children using barcodes or QR codes (Tanzania

and Zambia)

Background: Each facility had an

under-five paper register in which

all children within the catchment

area were recorded. This was for

the sole purpose of tracking

services offered to children. At the

facility level, HCWs generated a

child’s identification number with

the number sequence built upon

the provincial/district code, facility

code, serial number, and year code

for the year starting at one.

However, what was written in the

register and on under-five cards

was just the serial number and the

Photo: Bill & Melinda Gates Foundation/Riccardo Gangale


29

year, which was not unique at first glance, unless referencing the home facility written on the

card.

Solutions adopted: Due to this potential for discrepancy, the user advisory group(s) in Zambia

recommended a ZEIR code generated by the system to uniquely identify each child. To quickly

identify a child during a visit, QR codes were generated, printed, and adhered to the child’s

card. A random seven digits were generated and assigned to each child during registration. This

unique identification was correlated with the QR code attached to the child’s card. In Tanzania,

barcodes were used. Each barcode used a number containing ten random digits. The barcodes

were affixed to the child’s health card, scanned, and as the HCW registered that child by

entering the name and other information, the child’s record became associated with that

barcode in the EIR.

Lessons learned and challenges: It was challenging to identify a local vendor capable of printing

barcodes and QR codes in bulk, making it difficult for the government to procure these items

and maintain logistical sustainability. However, the team was able to find a local vendor to

produce the barcode stickers. For the barcodes to be read, the program tablets needed to be

equipped with a five-megapixel camera, adding to the requirements for tablet procurement. The

size of the barcode also impacted the success of the barcode scan. The camera most easily

read the barcode when it was larger, and adequate light was available. Lastly, as mentioned in

the Change Management section, the idea of barcode identification was not easily accepted by

parents, and some had negative reactions to their use.

Recommendations:

As technology improvements continue, more in-country vendors are being identified that can

provide tablets and barcodes. Continue to assess the local market for vendors that may

begin to produce and distribute items such as barcodes and QR codes.

In cases in which there may be supply disruption of the barcodes or QR codes, ensure that

the EIR also has unique identification in the system that can be seen by the HCW. In these

scenarios, the HCW can write the system identification on the child health card in the case

that the barcode or QR code stickers are not available.

Community birth registration (Tanzania and Zambia)

Background: Collecting data on children when they are born is critical for accurate recording of

children in need of immunization, also known as collecting an accurate denominator for service

delivery. In both Tanzania and Zambia, most births occur at a facility or hospital, but some births

still occur at home. Facility births are documented in paper registers; however, births that occur

at home often go undocumented, especially if the child is not brought to the facility during the


30

mother’s postnatal care. When children are unaccounted for, facilities cannot track all children in

the community who need lifesaving vaccines. This also has an impact on stock management

and reporting for immunization coverage, as it is difficult to plan for children not born in a facility.

Solutions adopted: In Tanzania, home births were captured and sent to the EIR via SMS using

the bring-your-own-device approach, as it was not cost-effective to buy equipment for

community leaders. The SMS notifications allowed the EIR to queue a child for immunization

based on the date of birth sent via SMS, and the child could be accounted for in his/her home

facility. However, given the low volume of home births and the high cost of training community

leaders, paired with the high volume of data entry errors, the SMS birth notification process was

cancelled in Tanzania. In Zambia, SMS functionality was not fully implemented because the

Ministry of Health requested that BID work with the United Nations Children’s fund (UNICEF) on

the SMS component to do this in the near future.


In Tanzania, data quality from community leaders using SMS was very poor and the

technology did not allow for built-in quality checks. Other available choices for technology,

like unstructured supplementary service data (USSD) and mobile applications, would have

been better suited; however, each has limitations, whether it be number of characters,

session time between navigation for USSD (as messages have to be sent back and forth

with USSD), or cost for mobile application hardware. Even after the incorrect information

was sent to the EIR, community HCWs would send an additional SMS with updated and

correct information. However, this created multiple records for the same child and duplicate

data in the registry.

It was difficult to manage even

the discounted rate for sending

SMS messages for health.

The team in Tanzania partnered

with the Registration and

Insolvency Trusteeship Agency,

which is developing initiatives to

provide birth certifications to

study the possibility of

integrating data from the EIR

and their system, and this will

help with the process of

capturing home births and using

that data in the EIR.

In Zambia, the team has been

working with UNICEF to use the



31

data from their mobile vaccination application (mVacc), which collects birth notifications.

This information would also be shared with the EIR to capture home births for a more

comprehensive picture of the target population for facilities.

Current SMS technology is insufficient to guarantee high-quality data. Data quality from the

field can suffer given lack of experience and training among community-level HCWs or

community leaders.

Recommendations:

Work with partners who are focusing specifically on home birth notifications and set out

technical plans for an integrated approach to the data being collected for the EIR.

Work closely with governments and partners to ensure that SMS is provided free of charge

or at reduced cost when being used for health equity.

Locking devices to conserve data bundles (Tanzania and Zambia)

Background: Tablets were distributed to HCWs to collect and enter patient data into the EIRs.

Initially, the tablets were not locked to prevent downloading of additional apps, to build trust with

HCWs. However, users added applications, which quickly depleted data bundles. BID Initiative

staff then installed software to prevent the download of applications.

Solutions adopted: In Tanzania, AppLock was installed on tablets to prevent downloads of

unnecessary applications. This prevented users from accessing the app store. In Zambia, Vault

App was installed on the facility tablets, which restricted access to certain Internet services,

such as YouTube, and web browsers.


One challenge to this solution was that AppLock prevented all downloads, including new

updates to the TImR software. The team adjusted by no longer locking the Play Store but

locking non-TImR application downloads.

Zambia faced other challenges with Vault App. Only five applications could be blocked free

of charge; blocking of additional apps required a paid subscription. Therefore, the team

locked the system settings to restrict access to certain applications. In rare cases, the facility

figured out the password and uninstalled the blocking application. In some instances, a user

simply removed the SIM card from the tablet, put it in their device, and used up the bundles.

We learned to involve the district in monitoring for abuse of data bundles, tied to

maintenance of clinical equipment.

In Tanzania, HCWs were able to access the locking password when requesting technical

support. Technical support employees inadvertently revealed the password when providing


32

assistance, and HCWs were able to unlock their devices. In some instances, HCWs were

able to uninstall the AppLock application.

Recommendations:

To protect data bundles, install software to prevent downloading of unnecessary apps.

Although free security apps can be used to protect data, it is important to select the correct

option.

Highlight the importance of respecting the data bundle. This was successful in mitigating

use of apps not necessary for normal program needs.

Involve district leadership in monitoring bundle utilization when conducting supervision visits

to facilities, which we found successful in creating accountability for data use among HCWs.

Consider enterprise mobility management (EMM) applications as a way to manage devices.

These tools can provide insight into which applications are running on a device, the version

of those applications, and when the devices are connected to the Internet. Although EMM

has a wide range of functionalities, most require an Internet connection.

Back-entry of data (Tanzania and Zambia)

Background: The initial BID Initiative strategy called for back-entry of children’s historical

immunization records in order to populate EIRs with children at the beginning or middle of their

immunization schedule. The purpose of back-entering data from current immunization records

at the facilities into the EIRs was to provide HCWs with a more efficient start-up when adopting

the new data collection tools. For example, rather than registering every child and entering all

the child’s past immunizations into the EIR during their first visit with the electronic system in

place, already back-entered data would allow the HCW to quickly search and confirm the child’s

correct immunization history and update the record with the services provided that day only.

Solutions adopted: Initially, Tanzania contracted with an external company to digitize the data

from the paper registers into the EIR. A PATH team member took photos of the paper

immunization register at the facility and uploaded the photos to a secure database. The photos

were digitized into an Excel spreadsheet that was then entered into the database to populate

the EIR. Due to the challenges detailed below, the team shifted the strategy to include data

entry staff at selected facilities and finally moved the back-entry process to HCWs, with support

from DDUMs, as children came to the facility. Zambia initially identified an external company to

back-enter information taken from photos of children’s under-five cards on a web-based data

collection platform. Once the data were cleaned, the system developers then exported the data

to the EIR. However, this approach did not yield the expected results because many records

were not being entered, leaving gaps in data. The strategy shifted to engaging HCWs to


33

conduct back-entry directly into the EIR; a small stipend was provided based on the volume of

records entered. Under-five cards were the source documents for this process as well.

Lessons learned and challenges: The process of

back-entering data was not as straightforward as

initially anticipated due to two main factors: the

technology used to convert paper data to digital data

was not as flexible as anticipated, and existing data

were low quality (especially in the paper register

books). In Tanzania, the system used to digitize

photos of immunization forms into the EIR database

required perfectly straight photos with legible notes.

The team went through many iterations attempting

to work within the system, including building photo

stands to steady the camera and immunization form,

but it took considerable time to render images that

worked with the system. In addition, the quality of

the data in the immunization register books was low.

Often, HCWs entered data first into the child health

cards kept by caregivers, then the tally sheets for

reporting, and finally the immunization book at the

facility. The team attempted to clean the data, so

the immunizations matched the schedule, but this became time consuming due to the large

quantity of records. Although the dates may have been correct, the schedule may have been

wrong in comparison to the national schedule. The decision was eventually made to discontinue

use of paper registers as the sources of data for back-entry in both countries. The team then

switched to images of the child health cards, given that these had the better-quality data.

However, the most efficient way to ensure the timeliness and quality of back-entered data was

for HCWs to enter the data directly into the EIRs. This proved more successful and allowed for

faster, more accurate back-entry of data. In Zambia, a success from this process was the shift

of data entry protocol. Originally, HCWs first entered data into the paper immunization registry

and then entered the data into the EIR. As governments look to digitize their health care

services and data, the relatively simple switch to having HCWs enter data into the EIR first and

the immunization book second (or not enter data into the books once policies are in place) will

increase use of the new tools by the HCWs, use of the data because it is in real time, and

supervision from the district since the data are readily available, and provide a catalyst for the

adoption of digital health solutions.



34

Recommendations:

When discussing back-entry of data, first identify the need for back-entry, or the benefits of

utilizing these data, including more efficient initiation of HCWs into use of the new tools

and/or having the data in the database for monthly implementation reports. This should help

to inform other specifics such as parameters around the historical data to be entered and

the time frame to be captured.

Choose the best-quality data source. Although Tanzania captured data from the paper

immunization register, Zambia opted to capture data directly from child health cards, given

their often higher quality of data, and the facility staff actually chose to enter records for all

children, not just for children younger than nine months.

Once the data source is identified, the back-entry of records can be time consuming. Start

with a few facilities with different populations of children due for immunizations (high volume

and low volume) to gather evidence on the average time needed to back-enter records

based on the criteria (e.g., nine months of data) as well as the quality of the data source

chosen to test the feasibility of back-entry, and then make plans based on what is learned in

the test.

Procurement

Background: The interventions identified by the countries to address their data challenges

included developing an EIR. In order for facilities to collect and use patient-level data, tablets

needed to be procured for the mobile application of the EIRs. Before tablets were procured,

software developers working on the EIRs provided hardware requirements and specifications

necessary for the system to function; these became the purchasing guidelines for BID staff.

Tanzania procured around 700 tablets for use and Zambia around 270 tablets.

Solutions adopted: Technology changes frequently, and our procurement team had to stay in

close connection with local vendors to ensure adequate quantities would be available for

procurement in-country, and items could be replenished.

Lessons learned and challenges: Both countries were able to procure tablets with an Android

operating system. However, team research showed that tablets compliant with the hardware

specified by the EIS software developers were not easily found in-country. This lack of

availability made it difficult for staff to find tablets that met all the standards of the EIRs and were

affordable and would be able to access the in-country network. For example, tablets procured

from outside the country were sometimes locked, thus unable to access in-country mobile

networks, meaning that to “break into” the tablets, key features would have had to be disabled.

These tablets were returned to the vendor.


35

Recommendations:

In initial discussions with the software developers about what specifications tablets should

have, share a list of the devices available for purchase locally to see if these options would

work. The BID teams approached local vendors to see if the demand for and quantity of

tablets would be enough to shift the procurement and availability in-country. This may be

especially relevant when scaling nationally, as the bulk purchase of items creates an

alternative opportunity for local vendors. Be sure to check the specific model number of

each tablet because this will show whether or not those tablets are locked to prevent access

on specific mobile networks. Before going to sites such as Amazon for procurement, check

regional hubs (e.g., Nairobi or Johannesburg) for hardware options.

Work closely with the Ministry of Health and the information and communication technology

group to understand thoughts and policies on procurement as well as to possibly negotiate

costs in the future.

Software Development Process

Change logs (Tanzania and Zambia)

Background: As bugs were documented during testing and launch of the EIRs, new Android

Application Package files addressing certain bugs were published by the developers. The most

thorough releases included change logs that captured which bugs were fixed and what new

features or functionality were included. If a change log was not included, it created confusion

when the team was unable to identify appropriate areas for user testing.

Solutions adopted: Change logs became required for every release in the development of both

EIR systems.

Lessons learned and challenges: As the systems increased in complexity throughout the

development process, ad hoc structures needed to be standardized to ensure appropriate

communication among the implementation team and the software developers.

Recommendations: Develop and adopt standards throughout the implementation process as

needed. Make sure to identify areas where protocols would be helpful and provide the

necessary support to implement them at the beginning of the process.


36

Defining system requirements (Tanzania and Zambia)

Background: As a way to document the requirements – functional (or technical specifications)

and non-functional (or operational specifications) – of both countries’ EIRs, Tanzania and

Zambia used the Public Health Informatics Institute’s Collaborative Requirements Development

Methodology (CRDM). CRDM documents who interacts with which data within the context of

clinical workflows as a way to understand how information systems can support those

workflows. Tanzania and Zambia put these requirements into requests for proposals and

supported the government in deciding on the vendors to develop the registries. Kick-off

meetings were held in both countries with developers, PATH staff, and the Ministry of Health

(comprising national, provincial, district, and facility staff from the user advisory group[s]) to

define and approve the requirements for the EIRs.

Solutions adopted: The user advisory group(s) with representation from all levels of the health

system and key test facilities was formed to advise on system requirements, definitions, and

clarifications throughout the process of developing the EIRs, as well as all other interventions.

This was done to ensure all appropriate stakeholders participated in defining the needs of the

government, the appropriateness of the interventions for the users, and the feasibility of the

implementation strategy.

Lessons learned and challenges: Field visits were conducted, which included interviews and

observations, to further understand and clarify some requirements. Later, as the system was

being implemented, other needs came to light and new requirements needed to be included,

which required trade-offs with less important requirements. For instance, in Zambia, initial

requirements did not include recording supplements like vitamin A and deworming, but as the

system was tested, it proved more efficient to include these, as this metric was normally

captured by HCWs at the same time as immunization data were collected. Collecting the

appropriate data elements for service delivery and reporting were critical for the register to

eventually replace the legacy paper system. Other requirements shifted as feedback from staff

and testing revealed a need to further accommodate existing workflows, such as adding growth

monitoring information for children who attended an under-five clinic for only a weight check.

Additionally, in Zambia, we learned that the OpenSRP immunization module that had been used

in Pakistan required a few adjustments to match Zambia’s country needs.

Recommendations:

When defining requirements, include high- and lower-level stakeholder staff to ensure a

balance of understanding that creates a more robust list of recommendations.

Take time to observe service delivery in the field, to better understand workflows and

address other staff difficulties that may come up when adopting the system.

https://www.phii.org/crdm

https://www.phii.org/crdm


37

During testing, ensure the system is exposed early to all levels of stakeholders identified

during establishment of requirements. The danger of non-inclusion is rejection of system

changes based on the lack of testing feedback.

Local support for software development is needed (Tanzania and Zambia)

Background: The different software development groups that partnered with PATH on the work

of the EIRs had teams based in Albania, Canada, Kenya, and Norway. Not having the lead

developers in-country made communication challenging given time zone differences as well as

contextual understanding of the local use cases. Additionally, when there were critical system

issues, the fixes were often delayed due to the time zone differences and limited capacity

outside of the software teams without resources located in the demonstration countries.

Solutions adopted: Mohawk (TImR developer) and Ona (ZEIR developer) partnered with in-

country software companies in Tanzania and Zambia. This allowed the team, users, and

Ministries of Health to have direct contact with a developer team in-country, as well as built in-

country capacity for sustainable system maintenance.

Lessons learned and challenges: By partnering with local software companies, response times

to bugs were more efficient given the shared time zone and context. This partnership also

strengthened capacity for digital health within the country, both for the software company and

for the Ministries of Health, as the local software company could train ministry staff over the long

term.

Recommendations: Identify local software companies with experience in digital health and a

desire to learn and involve them in the process early and throughout.

System and software updates (Tanzania and Zambia)

Background: When deployment first began, new Android Application Package files for the TImR

were sent to the BID team by email, and in Zambia, posted to a private software site. As more

facilities began to use this system, this process became difficult for the BID team to manage.

There were times that the software releases sent over email did not include version numbers,

making it difficult for the team to keep track of which facilities were using which version

(important when logging bugs).

Solutions adopted: The team opted to switch from email and private websites to posting the

software releases and updates to the Google Play Store.


38

Lessons learned and challenges: During an iterative development process, new software

releases can come as often as every two weeks. In the testing phase of the EIR, major changes

to the software were shared via email and private sites. While this worked well for accessibility

when only a few facilities were using the system, it still required the BID team to drive to the

different facilities to either collect tablets and bring them to the PATH office to update the

version of the system or take data bundles to the facilities and attempt system updates onsite.

This method was not a viable way for facilities to update and use the most recent software

versions when scaling. Once the system was stable, software releases were deployed a last

time through email or a private site. Now, with a focus on bug fixes in both Tanzania and

Zambia, more minor releases with these bug fixes are deployed through the Google Play Store,

making the system readily accessible to all facilities once data or Internet connectivity is

established.

Recommendations:

Identify early on the necessary procedures and protocols for seamlessly scaling programs.

Using the application purchasing site to roll out current product versions created better

version control and enabled a sustainable system for updates beyond the life of the program

implementation.

Consider EMM applications as a way to push updates to deployed devices. These tools can

provide insight about what applications are running on a device, the version of those

applications, and when those devices are connected to the Internet. Although EMM has a

wide range of functionalities, most require an Internet connection.

Development time from three to six months or longer

Background: In the initial request for proposals process for the first EIR in Tanzania, the BID

Initiative required responders to confirm that the software could be developed to have a product

ready for deployment within three months. In the initial EIR development process, we learned

that three months was not enough time for development, as the technology was not as prepared

to meet user needs as we had anticipated. Contributing factors included the challenges of

having a system that could host multiple users on a single Android device, syncing between

devices in the same facility, and offline functionality for mobile devices at the facility level.

Solutions adopted: The next request for proposals for the EIRs (the second one in Tanzania and

both processes in Zambia) provided a development timeline of six months to have a minimum

viable product ready for deployment.

Lessons learned and challenges: Countries are forward-thinking when it comes to the

functionality required of digital health solutions to improve data use and immunization service


39

delivery. Specifically, it was challenging to design a system that could host multiple users on a

single Android device, sync data between multiple devices in the same facility, and provide

offline functionality for mobile devices. We learned that the technology was not as far along as

we had anticipated and therefore needed more development time to meet the basic set of

requirements.

Recommendations: Ensure requirements are grouped into minimally viable product categories

and which system requirements can be worked on and easily updated later, during rollout. This

allows a balance of getting the product available to users within a reasonable period of time and

addressing bugs that arise with the product, but also provides space to do ongoing work to fix

bugs and strengthen the platform over the long term.

Deciding on a new system when the current system does not meet

requirements

Background: The BID Initiative partnered with the governments of Tanzania and Zambia to

define the functional and non-functional requirements for their EIRs and through the request for

proposals advised on which systems could best meet those requirements. Functional

requirements included activities such as HCWs recording the doses provided and the system

automating reports. Non-functional requirements included auditing, security, and the ability for

the system to exchange data with national-level reporting systems. Time frames were set at

between three to six months for development of the EIRs to meet a minimum set of the

requirements for a minimally viable product that could begin to be deployed and used.

Solutions adopted: Both Tanzania and Zambia refined the requirements based on lessons

learned and sent out new requests for proposals when challenges arose with the systems

initially selected. Tanzania kept the existing EIR in several facilities already using the system, so

developers could fix the backend challenges. Zambia pulled the initial system from the test

facilities, as it was only in a few at the time.

Lessons learned and challenges: The launch in Tanzania was paused when challenges with the

EIR were first identified. In Zambia, when challenges were found, especially with lack of

auditing, the system was withdrawn from the test facilities. Both countries modified their

requirements based on lessons learned from implementation and these requirements were

shared for Ministry of Health approval. The requirements were then shared through an updated

request for proposal process. Tanzania continued rollout of its EIR while backend issues were

fixed to make the system function more appropriately. Zambia moved forward by deploying

other data quality and use interventions in the first three districts while the new EIR was being

developed in parallel. As these changes happened, more frequent communications occurred

with both governments, the users, and the donor to ensure a critical path to success. We also


40

learned that without the EIR tool and its dashboards, it was difficult to show success due to lack

of data accessibility to the Ministries of Health to show the progress.

Recommendations:

Identify deal-breaker requirements such as lack of auditing and security issues. A good way

to navigate this is to pull the more difficult requirements into earlier development sprints and

milestones as a way to more quickly identify if the developers are capable of meeting these

requirements.

Continue ongoing testing with users and the Ministry of Health to assess system

performance (see User-centered design). This clinical and data use expertise can also

ensure in earlier stages whether development is on the correct track to meet the

requirements and data needs of the country.

Have a transition plan ready if system development fails. For example, in Zambia the data

quality and use interventions were deployed in parallel to system development. This allowed

the culture of data use to continue to mature so that when the EIR was ready, it was a tool

to complement the strengthened data use by HCWs.

Documents

BID Initiative Lessons Learned Encyclopediabidinitiative.org/wp-content/uploads/FINAL_Lessons...DDUM district data use mentor EIR electronic immunization registry EMM enterprise mobility