View
1
Download
0
Category
Preview:
Citation preview
The Social Observatory
Research to Improve Adaptation and Implementation
The “Theory”
CITIZENS/ CLIENTS
STATE
MARKETS CIVIL
SOCIETY Access & Accountability
Electoral & Social Accountability
Challenges
Repairing civil society and political failures is a much harder task that needs a fundamentally different approach to development
Variability of local context and the unpredictable nature of change-trajectories highlight the importance of developing effective systems of internal learning and monitoring
Such projects require constant adjustment, learning in the field, and experimentation in order to be effective
“Scaling-Up” is one of the biggest challenges - Interventions that work well with small populations routinely face serious challenges in expanding to a larger number of communities.
Mansuri and Rao, Localizing Development: Does Participation Work?, 2012
Hence Adaptive Capacity Social Observatory
Improving the project’s ability to see, learn, and adapt is critical for complex projects and for scaling up Development as Prozac and Development as Therapy
Lots of talk: Hirschman (1967), Rao and Walton (2004), Ellerman (2005), Easterly (2006), Woolcock (2009),
Mansuri and Rao (2012), Ramalingam (2013), WDR (2015), Matthews, Pritchett and Woolcock (2016)
But little action: World Bank VP for South Asia, Isabel Guerrero – Put up or Shut Up
2011: The SO set up as a joint initiative between the World Bank Research Department and the Bank’s South Asia Livelihoods team Leverage $4 per $1
Principles of the SO
“Embedded” Research Collaboration between research and project staff TTLs, Project Director, M&E in Charge, and grass roots
functionaries
Inter-disciplinary Question Drives Method(s) Team of economists, sociologists, management information
specialists, behavioral scientists
Objective: Research for better implementation Help projects build adaptive capacity
Our partners
$ 2 billion India Livelihoods Portfolio
Bihar – The JEEViKA Project
Tamil Nadu – Pudhu Vaazhvu Project
Livelihoods Projects: Women’s Empowerment and Poverty Reduction
CORE: Facilitated credit intervention. SHG mobilization. 7-12 women. 10-15 per village. Headed by a Village Organization
SHG Federation: Village-Block-District
VERTICALS: Think of SHGs as a highway. Roll out various anti-poverty programs, nutrition interventions, women centered interventions (about 30 verticals currently in operation)
GOALS: Women’s Empowerment, Poverty Alleviation, Building Sustainable Livelihoods
Adaptive capacity in practice
1. Long-Term Feedback: Mixed-Method IEs (Quant to understand magnitude of impact – “how much”), Qualitative (to understand mechanisms - “why”)
2. Everyday Feedback: Management Information Systems, Decision Support Systems, Process Monitoring
3. Citizen/Beneficiary Feedback: To give beneficiaries a role in improving design and implementation
LONG TERM FEEDBACK
Mixed methods evaluations in Bihar
Long-Term Feedback
6 IEs of the “Core Intervention” in Four States for External Validity
4 IEs of “Verticals”
Will focus on two sequential mixed-method evaluations of JEEViKA in Bihar to understand the added value of integrating qualitative and quantitative methods in evaluation
Project Timeline
Research Timeline
JEEViKA timeline
2006 2011 2014 2015 2016 2022
PSM Data
Phase 1 Phase 2 Phase 3
RCT Baseline
Qualitative Start
RCT Endline
Qualitative End12 cycles
375,692 741,847 2,908,010
Target
12 million
RCT Midline
Impact of JEEViKA in Phase 1: Propensity Score Matched on Project Selection Variables
Savings and Debt (Diff-in-Diff) Effect Size (Percent)
Savings 290.63
Percent HH with high cost loans( from 2008) -43.39
Amount borrowed (New loans) -46.72
Empowerment (Diff-in Diff) Effect Size (Percent)
Visit Panchayat Meetings 534.61
Visit local shop 21.54
Visit PDS 58.99
Visit Health Center 44.30
Visit Relative 37.08
Provide input on decisions on Children’s Education 36.65
Report having an opinion on politics 333.33
* Datta, Upamanyu (2015) , World Development , Volume 16
Why qualitative?
• How did the project change culture and social norms to help equalize gender relations?
– Sanyal, Rao and Majumdar (2015),“Recasting Culture to Undo Gender: A Sociological Analysis of the Jeevika Intervention in Bihar, India.”World Bank Policy Research Working Paper
• Subset of quantitative sample – 5 matched treatment (Phase I) and 5 control• 3 years, 10 villages, 12 cycles of data collection • 1 cycle = 200 interviews, focus group discussions and direct observation of
group activities, which amounted to 2400 transcripts. • Five ethnographers entered each village every quarter for a week
Qualitative: MethodologyDistrict Village No. of open-ended Interviews No. of
FGDs
Joiner Non-joiner Husband General Informant
Madhubani Phase I 120 48 24 24 24
Phase II 120 48 24 24 24
Control 168 24 24 24
Muzafarpur Phase I 120 48 24 24 24
Phase II 120 48 24 24 24
Control 168 24 24 24
Madhepura Phase II 120 48 24 24 24
Control 168 24 24 24
Saharsa Phase II 120 48 24 24 24
Control 168 24 24 24
Sub-themes Treatment Control
Act of borrowing Less humiliating and more dignity when borrowing from SHG versus moneylender
Considered begging; do not like borrowing or being rejected or defaulting with a moneylender
Ability to obtain a loan Depends on collective capacity to bargain
Depends on individual capacity to bargain
Decision on taking loans Made by women themselves
Women typically act on behalf of husbands
Information on village credit
Women are better informed on village moneylending networks and interest rates
Women lack village-wide information on credit networks
Economic themes
Sub-themes Treatment Control
Capabilities Women see themselves as capable of being active participants in public debate
Women see public sphere as ‘masculine’
Opinions on local govt. Women voice opinion on corruption and necessity of bringing change
Women seldom give theiropinion on a public forum
Problem-solving Jeevika women arbitrate among themselves, e.g. land conflicts, domestic violence.
Women’s issues are rarely taken up by themselves or in Aam Sabhas i.e. public forums
Fighting elections For Sarpanch, Mukhiya,ward members
Only proxy-Mukhiya i.e. on behalf of husband
Social and Political themes
JEEViKA gives women exclusive access to a set of physical resources, symbolic resources, and an institutional environment – all of which were perceived as ‘masculine’ prior to project
PHYSICAL RESOURCES
Group money, a passbook, a moneybox
SYMBOLIC RESOURCES
Creating an alternative identity for poor women that cuts across caste
Democratizing financial decision making on disbursement of loans, signature and financial literacy
INSTITUTIONAL ENVIRONMENT
SHGs, VOs, CLFs, rituals
Access to an alternative source of credit than moneylenders
How did change come about?
Culture is not an immutable constraint for development: can be undone by giving economically and socially disadvantaged women access to a well-defined network of peer-women and new systems of ‘knowledge’
Norms can be changed in a short period of time: a re-iterative process of collective violation of behavioral injunctions on women is key (Butler 2004)
How JEEViKA alters deeply entrenched social norms
Impact of JEEViKA in Phase 2: RCT with Pre-Analysis Plan*
Randomized Phase-in across 7 project districts Evaluation sample: 9000 households in 180 Villages
90 villages randomly assigned to project treatment Project did not know which villages were part of the evaluation
sample
Baseline Survey- 2011
First follow up - 2014 Exposure of around 2.25 years to the project
Second follow up scheduled for 2016
*Datta, Hoffmann, Rao, Surendra, “Report on the Impact of JEEViKA: Evidence from a Randomized Rollout,” November 2015
Impact of JEEViKA: Phase 1 Vs. Phase 2
Phase 1: PSMDiff-in-Diff
Phase 2: RCTANCOVAestimates
Savings and Debt PercentageChange
Percentage Change
Savings 290.63 60.02
Does household have any high cost loans -43.39 -7.48
Total high cost Debt -46.72 -15.15
Empowerment
Visit Panchayat Meetings 534.61 Not Significant
Visit local shop 21.54 Not Significant
Visit PDS 58.99 Not Significant
Visit Health Center 44.30 Not Significant
Visit Relative 37.08 Not Significant
Provide input on decisions on Children’s Education 36.65 Not Significant
Report having an opinion on politics 333.33 Not Significant
* Due to the retrospective nature of the PSM, some variables were defined in slightly different ways
Why the difference in results?
1. Difference in methodology
2. Shorter time lines:
Five years in Phase 1, and two years in Phase 2
3. Difference in implementation quality
Common Knowledge:
A. Doubling coverage
• Large number of new staff hired
B. Poor decision support systems to manage expansion: o Proper MIS not set up
o Poor process monitoring
Insights from our qualitative research– Differences in the quality of project facilitation between Phase
1 and Phase 2
What went wrong with implementation in Phase 2?
Phase I Phase II
Doing a thorough power analysis / informal information gathering
Getting ‘buy-in’ for the project
Social mapping as a means of taking the site of knowledge production to the village
Social mapping is done as a means of arriving at a number of target households
Making self-help the end goal Making jobs or lower interest rates the end goal
Turning first movers into ‘eyes and ears’ of the community
Turning first movers into ‘eyes and ears’ of the facilitators
Initial Mobilization
Phase I Phase II
Ritualization / ‘performing’ of participation
Rituals are seen as burdensome
Community ownership over the project is taken literally
Community ownership over the project is rhetoric
Engaging head-on with the messy business of preventing elite capture
Keeping community politics at bay
Enrolling a nexus of supporters through the project’s life cycle
Building support is limited to the beginning of the project
Group Meeting Stage
Qualitative evidence critical in interpreting quantitative results
Decision support systems for everyday learning are essential to manage expansion/scale-up
Next element of our work on building adaptive capacity
Learning and Adapting from Evaluation
EVERYDAY FEEDBACK
Supporting Grassroots decision making in Bihar
Everyday Feedback
– V vs Tepee /\
– Process Monitoring Systems
– Decision Support Systems for JEEViKA’s Core intervention (Huge Challenges)
– Tracks 3 million women
– Dashboards at Every Level
– Community Based Nutrition Intervention Tracking
CITIZEN FEEDBACK
Participatory Tracking in Tamil Nadu*
*Palaniswamy, Rao, Sakhamuri, Shajeevana, Xia,“Democratizing Data: Participatory-Tracking in Tamil Nadu, India,” (forthcoming)
Modernize PRA with new technology and methods
Democratizing Data
Census of program participants
Pilot of 32000 women in PVP
Government has requested an extension to 10 million women in Tamil Nadu
Origin
Step 1: Develop Questionnaire
Community Based
Uses women’s networks
Tested by community members with community members
Finalized questionnaire should take no more that 30 minutes
What makes this questionnaire different?
Overlap in themes covered NSS: 17 %
LSMS type survey: 70 %
Covered a range of themes: Livelihoods, Economic Welfare, Food Security and
Nutrition, Empowerment, Access to public services and programs, Political Participation
Differed in framing and emphasis
A sample of questionsFood and Nutrition
How much do you spend on the purchase of vegetables in a month?
Does the person who eats last get enough to eat?
Marriage
What was your age at the time of your marriage?
Was your decision taken into account at the time of your marriage?
Did you marry your relative?
Empowerment
Who makes decisions on assets and loans in your family?
Do you decide on what clothes to wear based on your own preferences?
Have there been any instances of violence against women in your village?
Digital Participation
Can you use a mobile phone on your own?
Can you read and send text messages?
Step 2: Data collection and management
Participatory
Implemented and managed by CBO members
PVP project staff- Coordinating role
Other Key features
Tablet based
Data Quality and Validation
Designed for users with low digital literacy
Step 3: Data visualization
Empower respondents to analyze and act on their own data
Pilot PVP: Data visualization
Pilot PVP: Data visualization
Pilot PVP: Data visualization
Step 4: Data Feedback
Feedback
• Village Planning
• Gram Sabha –Deliberative Decision Making
• Planning Village Organization Budger
• RCT - half the villages randomly received treatment
• PVP had problems unfortunately
• Yet, results show visualized data have better results
• Expansion to rest of Tamil Nadu
• Planning for Jharkand
Some Uses of Participatory Tracking
An alternate citizen narrative of poverty and well-being
High frequency census data
Public goods decision making
– No incentives for project managers to acknowledge or learn from failure
– No incentive to tolerate pesky researchers
– Requires a stable project with an engaged team and Project Director
• So personnel changes can be a stumbling block
– Funding is a big issue – who pays?
Challenges
Projects have been very open and brave to allow us to observe them from such close quarters
Extremely difficult to get implementation right in the field, especially for complex, large scale projects
Embedded approach has been critical to the successes we’ve had
Improving the adaptive capacity of projects by changing the culture of implementation is difficult but can be done
Conclusion
SO Team
Core team: Vijayendra Rao (Head)
Nethra Palaniswamy , Upamanyu Datta, Shruti Majumdar, Smriti Sakhamuri, Samrat Ghosh, Nandini Krishnan, Madhulika Khanna, Nishtha Kochhar, G. Manivannan, Pankaj Varma
WEBSITE: http://socialobservatory.worldbank.org
+ 15 Academic Collaborators
+ 13 Implementation Partners
Recommended