Embed Size (px)
Citation preview
MFLN Evaluation and Reporting
Developmental Evaluation
“DE supports innovation development to
guide adaptation to emergent and dynamic
realities in complex environments.” —
MQP
Goal: support project, program, and
organizational development with timely
feedback
Core DE Question
What is getting developed and what are
the implications of what gets
developed?
Complexity theory of change:
Bring people together who are
knowledgeable and committed and they
will self-organize, take action, and work
together to create movement, innovation,
and change.
Key Characteristics of DE
Focus on development
Focus on development
Development is when people are
changing what they are doing, and the very
nature of the standards are also changing.
Standards are adapting to changing
conditions.
NOT the same as continuous improvement
(formative eval or accountability)
– Quality improvement is helping programs
meet standards that have been set.
Complex and dynamic
environment
Multiple leadership
nodes
Multiple paths
forward
Rapid and continual
feedback
Ask evaluative questions
Apply evaluative logic
Gather and report evaluative data
Evaluation becomes part
of
innovation/intervention
Developmental Evaluator
Works collaboratively with innovators
to conceptualize, design, and test new
approaches
in a long-term, ongoing process
of adaptation, intentional change, and
development.
Evaluation for MFLN
• Ongoing
development
• Innovation in
learning
– Internal
practice
– External
product
• Reflection and
action
DE for MFLN
MFLN: Focus on development
Programming has been established…
– Identify guiding principles that inform ongoing
development
– Establishes track record, reliability, and
reputation
... BUT is not necessarily fixed
– Innovate, innovate, innovate
– Internal adaptability to meet changing external
standards and conditions
Cooperative agreement
environment
+
eXtension Initiative
Cooperative Agreement
“An opportunity to provide responsive and timely educational programming through being actively nimble, flexible, innovative, and creative in true partnership with our funders as they identify organizational priorities and ask us to engage in the construction of appropriate and necessary deliverables that meet the on-time needs of the target audience. The MFLN and funding partners are seen as true and honest equals in expertise, providing valued and accepted feedback bi-directionally. This often results in surpassing expectations for success in
Constant evaluative
feedback and discussion
Monthly reports (internal, DoD)
Quarterly reports (NIFA)
Annual reports (internal, DoD, NIFA)
Webinar evaluation reports (CAs)
Quarterly webinar reports (internal, DoD)
Weekly leadership team meetings
Social Media Specialists
Instant
messaging
Hangouts
Google docs
Phone
Internal evaluator
Member of leadership team
Elucidates innovation and adaptation
processes
Tracks implications and results
Facilitates data-based decision-making
DE for MFLN: What It Looks
Like
Continuous
adaptation
DE for MFLN: What It Looks
Like
Continuous
adaptationSocial, political, economic, technological, and
demographic patterns
DE for MFLN: What It Looks
Like
Continuous
adaptationSocial, political, economic, technological, and
demographic patterns
Emergent developments/needs of our target
audiences
DE for MFLN: What It Looks
Like
Continuous
adaptationSocial, political, economic, technological, and
demographic patterns
Emergent developments/needs of our target
audiences
Cooperative agreement environment
DE for MFLN: What It Looks
Like
Continuous
adaptationSocial, political, economic, technological, and
demographic patterns
Emergent developments/needs of our target
audiences
Cooperative agreement environment
eXtension
DE for MFLN: What It Looks
Like
Continuous
adaptationSocial, political, economic, technological, and
demographic patterns
Emergent developments/needs of our target
audiences
Cooperative agreement environment
eXtension
DoD
DE for MFLN: What It Looks
Like
(not this. . . .)
DE for MFLN: What It Looks
Like
(but this. .
. .)
Inspired by Jeff Conklin, cognexus.org
Time
Complex development situations are ones in which this…
25Michael Quinn Patton
AEA eStudy webinar 2014
Michael Quinn Patton
AEA eStudy webinar 201426
And this…
Turns out to be this…
Time27
Michael Quinn Patton
AEA eStudy webinar 2014
Michael Quinn Patton
AEA eStudy webinar 201428
…looks like this
DE for MFLN: What It Looks
Like
Inductive
DE for MFLN: What It Looks
Like
InductiveChaotic and messy
DE for MFLN: What It Looks
Like
InductiveChaotic and messy
Embrace forks in the road
DE for MFLN: What It Looks
Like
InductiveChaotic and messy
Embrace forks in the road
Developmental moments
Track what’s going on
Understand what’s going on
Adapt to what’s going on
DE for MFLN: What It Looks
Like
Everyone is an
evaluator
DE Successes for MFLN
DE Successes for MFLN
Very responsive to our partners’
programmatic requests and innovations
DE Successes for MFLN
Very responsive to our partners’
programmatic requests and innovations
You. Here. Now.
DE Successes for MFLN
Very responsive to our partners’
programmatic requests and innovations
You. Here. Now.
Programming
DE Successes for MFLN
Very responsive to our partners’
programmatic requests and innovations
You. Here. Now.
Programming
“Walk the talk”: model use of social media,
collaborative learning, personal learning
networks
DE Challenges for MFLN
Innovation is challenging, slow, and can be
contentious
DE Challenges for MFLN
Innovation is challenging, slow, and can be
contentious
Moving from program improvement mode to
redesign based on evaluation findings
DE Challenges for MFLN
Innovation is challenging, slow, and can be
contentious
Moving from program improvement mode to
redesign based on evaluation findings
IRB challenges
DE Challenges for MFLN
Innovation is challenging, slow, and can be
contentious
Moving from program improvement mode to
redesign based on evaluation findings
IRB challenges
Daily, evaluative thinking
DE Challenges for MFLN
Innovation is challenging, slow, and can be
contentious
Moving from program improvement mode to
redesign based on evaluation findings
IRB challenges
Daily, evaluative thinking
Platform delivery limitations
DE Challenges for MFLN
Innovation is challenging, slow, and can be
contentious
Moving from program improvement mode to
redesign based on evaluation findings
IRB challenges
Daily, evaluative thinking
Platform delivery limitations
Military culture
DE Challenges for MFLN
Innovation is challenging, slow, and can be
contentious
Moving from program improvement mode to
redesign based on evaluation findings
IRB challenges
Daily, evaluative thinking
Platform delivery limitations
Military culture
Scaling up
DE Features for MFLN: Nodal
EvaluationsWebinars
Social media
Focus groups
Deliverables/plans of work
• Why as much as what
Documentation
• Change, innovation, decision-making
DE Features for MFLN:
ReportingFrom Concentration Areas
• PIs
– Monthly via Google forms: narrative updates,
pubs, presentations, challenges
• Social Media Specialists
– Monthly via Google forms: blogs, AaE, CEUs
– Webinar evaluation reports
DE Features for MFLN:
ReportingTo CAs:
• Monthly social media from Sprout Social
• Quarterly webinar data reports
Internal/External:
• Monthly (internal/DoD)
• Quarterly (NIFA)
• Quarterly webinar (internal/DoD)
• Annual (internal/D0D/NIFA)
DE Features for MFLN: Reflective
Discussion
CAs:
• Monthly social media specialists meetings
– NetLit, processes, evaluation,
communications
Leadership:
• Weekly meetings
PIs:
• Monthly meetings
DE Features for MFLN: Learning
and Working Paradigms
Leadership
Reflection
Transparency
Collaboration
Action
Innovation
