Guide to Monitoring eLearning Programs · localization of the tools. Although some research activities will ... Guide to Monitoring eLearning Programs 5 Step 1: Establishing the Setting

Guide to Monitoring eLearning ProgramsIntel Education Transformation Standard Research Design and Toolkit

2 Guide to Monitoring eLearning Programs

2

Table of Contents

Design Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

Toolkit Summary Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Integration Context Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Integration History Interview Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Program Report Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Research Customization Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

School Leader Interview Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Information Technology Coordinator Interview Protocol . . . . . . . . . . . . . . . . . 30

School Tour Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Community Learning Center Staff Interview Protocol . . . . . . . . . . . . . . . . . . . . 35

Teacher Interview Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Classroom Observation Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Student Interview Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Home Use Interview Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Data Triangulation Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Qualitative Report Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Intr

oduc

tion

Intel Standard Research Design and Toolkit: Introduction

This document presents an overview of the Intel Standard Research Design and Toolkit, outlining the broad contours of the suggested research activities that are intended to:

• Identify key success factors and challenges or limiting factors for educational technology integrations of Intel® technology-based solutions

• Recommend best practices, course revisions, and approaches to enhance education technology integration

• Substantiate policy recommendations for broad-scale 1:1 technology implementations and integrations

The research will provide insight into the progress and outcomes of the technology initiative, enabling program leaders to:

• Highlight program successes

• Generate support for the program amongst local stakeholders

• Identify areas and strategies for mid-course improvements to the integration initiative

Guide to Monitoring eLearning Programs 3

This Intel Standard Research Design and Toolkit was updated from the original toolkit based on valuable input from SRI, EDC, and field researchers Petar Nikoloski (Macedonia/Bosnia), Alejandro Artopoulos, Universidad de San Andrés, Dr . João Paiva, and Luciano Moreira Faculty of Sciences – University of Porto, Portugal, Dr . Chang Lee Hoon, Sultan Idris Education University Tanjong Malim, Perak, Malaysia, Alper Tolga Kumtepe and Cengiz Hakan Aydin Anadolu University, Turkey .

It is intended to be adaptable to multiple settings, levels of effort, and capacities of research teams .

If required, consultative services and training can be provided by Intel for local researchers to help them develop data collection and analysis skills appropriate for working with qualitative data from interviews and observations . For more information about the training, please contact lara .patil@intel .com (Lara Patil) or joya .chatterjee@intel .com (Dr . Joya Chatterjee) .

Design Overview

Our suggested research activities are in the following steps (Exhibit 1):

• Establishing the setting, which focuses on gathering background information about both the integration context and aspects of the rollout.

• Developing an outcome based evaluation and reference guide.

• Customization of the integration research, whereby local researchers draw from the Standard Integration Research Design and Toolkit to select data collection activities and modify the data collection instruments.

• Participant recruitment, which is done using techniques that are congruent with the size and the purpose of the integration research in any given setting.

• Data collection, which entails research activities (primarily interviews with participants and observations of technology use) to better understand the enactment of

Design O

verview

the integration and the changes that result in policy and practice.

• Analysis and reporting, in which local researchers use a variety of analytic and reporting templates to synthesize data, identify critical success factors and course corrections, and draw connections between enactment and change; this step draws from interim findings reported to Intel and may also include a comparative global roll-up.

• Post-analysis, which involves iteration of the protocols and some localization of the tools.

Although some research activities will be common to all educational technology integration settings, others will apply only to selected settings, warranted either by the nature of the integration or other parameters of integration research (such as resources available and local research capacity) . Exhibit 1 indicates essential activities; activities appropriate for some contexts,


Step

EssentialActivities

Activitiesfor SomeContexts

Activitiesfor GreaterDepth

Collect settingbackgroundinformation

Customize research plan for the setting and characteristics of the integration effort

Recruit high-levelstakeholders

Interview schoolor communityleaders

Interview teachersor communityinstitution staff

Observe classes or activities

Interview students

Compile and report qualitative findings

Reflect on studysuccess

Iterate toolkitTriangulate findings

Synthesize keyconclusions

Analyze mediamentions

Follow up interviews

Make comparisonsacross integrations

Select schools or community institutions for in-depth study

Select participants from among teachers or staff and students

Interview IT coordinators (where the role exists)

Identify rolloutcharacteristics

Interview Intelstaff

Interview high-level stakeholders

Research priorICT initiatives inthe setting

Begin trackingmedia mentions

Customization Recruitment Post AnalysisEstablishing theSetting

DataCollection

Analysis andReporting

INTEL EDUCATION TRANSFORMATION RESEARCH TOOLKIT TIMELINE OF RESEARCH ACTIVITIES

as determined by the size and the level of effort of the technology integration; and activities for greater depth, typically to be pursued in settings where additional resources and supports are available . The sections that follow describe the essential research activities in detail; suggestions for optional activities are highlighted in the sidebars .

The research typically occurs in phases, corresponding to increasingly deep levels of implementation . The exact breakdown of phases and activities will vary by country, context, and program . The following definitions, however, generally describe the depth of imple-mentation during each phase of the technology integration (Table 1) .

All of the research steps described above (from establishing the setting through post analysis) can occur at each phase of the research, creating an ongoing research cycle . However, some research steps will be more

relevant—or more of a central focus—at certain phases of implementation than others . For example, establishing the setting, customization, and recruitment are all particularly relevant during Phase 1, and researchers may only revisit these steps briefly during subsequent phases .

The instruments included in this Toolkit are useful across implementation phases and across research steps . The Toolkit includes instruments for gathering contextual information (Integration Context Form, Integration History Interview) and for collecting school- and classroom-level data (interview protocols for the school leader, information technology coordinator, community learning center staff, teachers, and parents/students; a school tour checklist, and a classroom observation protocol) . The Toolkit also includes guidelines for researchers related to customizing research plans and instruments (Research

Exhibit 1. Timeline of Education Transformation Research Activities

Customization Guidelines) and for conducting analysis and reporting (Data Triangulation Matrices, Qualitative Report Template) .

The technology integration research design emphasizes qualitative methods and an ethnographic orientation toward the research . Protocols and reporting tools are structured with enough flexibility to allow themes to emerge inductively from data to the greatest extent possible . Local researchers carrying out the integration research could come from a variety of disciplines and backgrounds . Ideally the expertise of local research teams should include a familiarity with both local education frameworks and policy contexts as well as a comfort with ethnographic inter-viewing methods (for example, semi-structured interview techniques that allow the priorities and the concerns of interviewees to emerge clearly) and inductive, bottom-up analytic approaches .

Des

ign

Ove

rvie

w


Step 1: Establishing the SettingEducational and political leaders around the world have recognized the urgent need to prepare their youth for the demands of the 21st century . This preparation includes the use of technology in support of learning, innovation, and active participation in social and economic development . These leaders share the understanding that the goal of equipping youth with 21st century skills involves the transfor-mation of formal and informal educational systems and the opportuni-ties available through them, often by integrating technology solutions with new approaches to teaching and learning .

In many countries, this goal calls for extensive reform of education systems . In each country enacting reform of this nature, educational change happens within—and is therefore shaped by—a local ecosystem of influences. These include the structure, characteristics,

5

orientations, and capacities of the existing education system; national and regional policies; school-specific professional cultures and capacities; and other social, institutional, cultural, and economic factors that transcend the boundaries of school .

Integration Context DescriptionsGathering background information about the setting in which the integration will occur forms the foundation for an integration research agenda seeking to explore how educational reform is (and can be) supported by specific technology solutions across diverse settings . Critical background includes information about the context in general, with particular emphasis on issues of education policy and practice, as well as details about the technology integration strategies .

The Integration Context Form, designed to address big-picture issues in educational and social change, will be completed by in-country researchers in each integration setting . As the primary source for gathering initial data about the

Step Customization Recruitment Post AnalysisEstablishing theSetting

DataCollection



DataCollection



DataCollection



DataCollection



DataCollection



DataCollection


ANALYSIS OF MEDIA COVERAGE

If the research budget and the

capacities of the local researchers

allow, we recommend that local

evaluators synthesize information

from current print and visual media

sources on the issues, sensitivities,

commonly held positions, and

rhetorical stances related to the

Intel® technology-based solution

in the integration environment.

Although we mention it here as

an activity for the early stages of

integration research, the process

of tracking and analyzing media

reporting could be ongoing through-

out the integration research. In later

stages of integration research, the

analysis of media coverage will

highlight prevalent attitudes and

public opinion about the technology

integration, and their influence, if

any, on the implementation.

Design O

verview

Table 1 . Phases of Implementation and Research

Phase Focus of Activities Research Focus

Phase 1: Vision

Lay the groundwork for the technology integration, including policy and funding . Work with schools and communities to generate support, determine existing levels of technology infrastructure and professional capacity, etc .

Policy-level context and groundwork for the project .

Phase 2: Planning

Engage participating jurisdictions; negotiate roles and communication channels; upgrade infrastructure and distribute technology . Includes initial implementation support, including professional development enabling teachers to begin experimenting with the technology in their teaching .

Infrastructure to support implementation, such as smooth technology distribution to participants . Also, implementation factors, such as whether teachers are getting the PD and technical support they need to im-plement the project successfully .

Phase 3: ImplementationFull program implementation, with participants using the technology as a regular part of their classroom or home experience .

Classroom practices; use of technology for teaching, learning, and home use; impacts for teachers, stu-dents, families, and communities .


FOLLOW-UP STAKEHOLDER INTERVIEWS

If the scope of the integration research allows, high-level stakehold-ers will be interviewed again toward the end of the integration research .

This second round of interviews will offer an opportunity to revisit the topics of the first but from an implementation perspective and to discuss the differences between design and enactment, local adaptations and other emergent changes, successful elements of the integration, and the lessons learned .

In addition, the second interview will address the media visibility of the integration, the issues related to managing the public’s expectations, and the extent to which public opinion influences, and is influenced by,the integration .

integration context, the Integration Context Form will provide researchers with important background information for understanding and analyzing various facets of the technology integration .

Key components of the Integration Context Form are:

• The structure, characteristics, and capacities of educational and social systems that are affected by the technology integration

• The educational, economic, and digital inclusion policy context

• The current capacities of the school system, especially with respect to teachers’ use of technology to support instruction

• ICT policy and infrastructure

Although the Integration Context Form consists of details local researchers are likely to be familiar with, the form and accompanying instructions will encourage researchers to refer to and cite other sources (government web sites, published literature, etc .) to ensure that the information is current, balanced, and accurate .

Integration History InterviewsInformation about the characteristics of the rollout will be obtained from the Integration History Interviews with implementation stakeholders . These include individuals responsible for making decisions about the size and nature of the integration, such as government representatives, university researchers, and members of the Intel sales team .

Whenever possible, initial interviews with local individuals may also help identify other implementation stakeholders perceived as the driving forces behind the integration and reveal the nature of the shared understanding between the policymakers and these individuals with respect to the direction and purpose of the integration .

Implementation stakeholders will inform researchers about goals of the rollout, the alignment between the goals of the integration and the goals of social or educational reform initiatives, the criteria for determining various aspects of the rollout, and the anticipated outcomes .

Specific topics to be covered in the interviews include the following:

• Education, social, and economic policy context

• Description of current strategic educational, social, or economic initiatives

• Goals for the specific technology solution being deployed

• The settings (school vs . home or community, grade levels, subject areas, etc .) for the integration

• Vision for the integration

• Place of the technology integration/ technology solution in the socio- political context

• Anticipated indicators of a successful integration effort

• Anticipated challenges and strategies to address them

Data gathered in this manner will indicate the designed characteristics of the rollout, the planned professional development and technical support, and the manner in which the Intel technolo-gy-based solution meets the needs of the integration setting . These data will set the stage for understanding the educational and social policy context, the extent to which the goals of the technology solution align with the direction of educational or social reform in the integration context, and the measures and metrics that will be used to evaluate the technology solution . All integration research efforts should include at least one round of interviews with Intel sales staff and high-level stakeholders at the beginning of integration research, ideally before or during the initial stages of integration .

Des

ign

Ove

rvie

w


Step 2: Customization of the Research Design

Local evaluators are encouraged to draft a research plan, which will involve customizing the Standard Research Design and Toolkit presented here for the specific integration context (including identification of research participants, recruitment strategies, data collection activities, and analysis and reporting approaches) . This process is supported by Research Customization Guidelines in the toolkit . These guidelines support local evaluators in using their background knowledge of the integration effort and the information gathered in the Integration Context Form and the Integration History Interview to modify the research design and data collection activities to make them relevant to the local

integration context . In addition to providing a blueprint for the integration research, the customization guidelines also aid in the selection and customiza-tion of data collection instruments . Although the goal of customization is to make the data collection instruments, especially interview questions, relevant for specific settings, local researchers are encouraged to retain the overall goals and high-level categories of the instruments so that the integration research remains comparable across different contexts.

During the customization step, researchers are encouraged to work with Intel (as appropriate) to develop a communication plan that allows for formative feedback regarding critical success factors and the possible need for course correction . Elements of the communication plan will include frequency of meetings and specification of a process that will allow local evaluators to discuss interim consider-ations and findings with Intel.


DataCollection



DataCollection



DataCollection



DataCollection



DataCollection



DataCollection


Design O

verview


Step 3: Recruiting Participants

Selecting groups of participants will depend in part on the planned characteristics of the rollout .

For example, an integration that focuses on schools will include teachers and students but may not include the coordinators of community learning centers or the individuals concerned with home use . An integration that focuses on enhancing learning across settings, however, is likely to feature home use interviews and interviews with coordinators of community learning centers . Notably, some participant groups (like implementation stakeholders) will be a part of integration research in all settings, regardless of the specific parameters.

Information about the country contextual factors and rollout charac-teristics, along with considerations for research customization, can be used to:

• Determine what kinds of sampling, recruitment, and selection will lead to the right amount of diversity of input and thought

• Identify high-level stakeholders

• Identify research participants

• Develop strategies for recruiting research participants that are appro-priate for the integration context and the specific goals of the integration research

Local considerations, such as the size of the integration research project and the purposes of the research, will shape the approach for selecting and recruiting research participants . Researchers in the field will use their understanding of the relevant categories of participation to develop the criteria for identifying groups of participants and the specific individuals who will participate in the research . Ideally, participants will be able to highlight the range of typical experiences with the Intel technology solution; in instances where the technology integration effort spans a broad range of contexts or a larger number of individuals, a broader, more diverse group of research participants will be recruited .

High-Level Stakeholder Selection High-level stakeholders (typically, national and regional leaders who play a key role in the selection and imple-mentation of the Intel technology solution) constitute the first group of participants after Intel staff to be interviewed as part of the integration research activities . In most instances, high-level stakeholder participants will be recruited through a snowball sampling process: Interviewing will start with one or two policymakers who are deeply involved in the technology integration; each policymaker in turn will be asked for his or her recommen-dations for individuals who should be interviewed . The total number of high-level stakeholders interviewed for the integration researchwill usually not exceed five people.


DataCollection



DataCollection



DataCollection



DataCollection



DataCollection



DataCollection


Des

ign

Ove

rvie

w


School or Community Institution Selection When the integration involves distribu-tion of devices in school settings, schools should be selected as the institutional centers for the data collection . When other institutions such as community centers or homes are central to the technology integration effort, sites should be recruited from among these institutions .

To the extent possible,1 the overall sample of schools, community centers, or homes should represent the range of settings, economic conditions, languages, cultural groups, and any other character-istics that are likely to illustrate typical experiences with the Intel technology

solution . In this research, participants will be grouped in a nested manner, meaning that students will be selected from classes taught by participating teachers from selected schools or from within groups of participants in other settings . Because school or community center selection will define opportunities for teacher and student recruitment, it must therefore take teacher and student characteristics into account . Also, the school leader and/or the IT coordinator should be available for interviews at each school .

Teacher and Student Selection Researchers will ask school or community center leaders or IT coordinators to identify candidate teachers or staff for

Design O

verview

interviews. Similarly, teachers or staff will be asked to support researchers in identifying student participants to be interviewed . Home use interviews, if they are applicable, will be conducted with these students . If researchers intend to carry out home interviews, family agreement to participate in home visits should be a condition for selection for student interviews . This will allow researchers to interview the same students in both home and school settings . In sum, whether the integration focuses on school, community, or home settings, recruitment will target a representative range of participants to the extent possible .


Step 4: Data Collection Step 4 includes a variety of activities intended to gather information to help researchers understand the enactment of the integration on the ground and the changes in attitudes, teaching and learning behaviors, learning environment, and policy that results . Exhibit 2 provides a detailed mapping of the main constructs of the research to the essential data collection activities .


DataCollection



DataCollection



DataCollection



DataCollection



DataCollection



DataCollection


Country

Language

Socioeconomics

Literacy Levels

Political Climate

Institutional Context

Organization and Structure

Goals

Characteristics

Capacities

Policy Background

Educational Reform Initiatives

Rollout Characteristics

Size

Distribution

Resources (PD and Support)

Strategies

Enactment

Professional Development

Technical and Other Support

Use Patterns

Change

Attitudes

Teaching and Learning Behaviors

Learning Environment

Policy

Exhibit 2. Sources of Data Required for Education Technology Integration Research

Inte

grat

ion

Con

text

Fo

rm

Inte

grat

ion

His

tory

In

terv

iew

s

Scho

ol L

eade

r In

terv

iew

s

IT C

oord

inat

or

Inte

rvie

ws

Scho

ol T

our

Che

cklis

t

Com

mun

ity L

earn

ing

Ce

nter

Sta

ff In

terv

iew

s

Teac

her I

nter

view

s

Cla

ssro

om

Obs

erva

tions

Stud

ent I

nter

view

s

Hom

e U

se

Inte

rvie

ws

CON

TEX

TIN

TEG

RAT

ION

IMPL

EMEN

TATI

ON

Des

ign

Ove

rvie

w


Immediate Data Processing After collecting data, researchers should plan to clean and process their data as soon as possible . For interviews, researchers should plan to synthesize details of the interviews they conduct as soon as possible after each interview, elaborating on detailed notes and noting areas for exploring broad themes within the interview . If interviews are recorded, researchers may consider transcribing selected parts then for later inclusion as quotations in reporting documents . For observations, researchers should plan to complete any unfinished portions of the observation protocol right away . Researchers should also note as many descriptive examples and quotations from the observation as possible, to help recall the specifics of the observation during later analysis, and to provide evidence for the data on the protocol .

Descriptions of Research InstrumentsHere we describe each of the data collection instruments included in this toolkit .

Interviews with School or Community Center Leaders2

Researchers will use the Student Interview Protocol, Protocol to interview administrators at sites selected for study . The same protocol may be used, with some modifications, to interview administrators in non-school settings as well . These leaders of schools or community learning centers will provide researchers with information about the school or community center site, teachers and students, and the support available for technology users . Their experience may reveal how demands and expectations from the policy level interact with teacher or staff and student needs at the local

Data Collection by Technology Integration PhaseAt each phase of implementation, the research must capture the current implementation context, the program activities currently underway, promising features and/or successes and challenges, and (for subsequent phases) what has changed from before. Some of the research instruments in this toolkit are specific to a particular phase of implementation. For example, the integration context form is likely to be most relevant at the very beginning of a technology integration effort (Phase 1), while the classroom observation protocol will only be useful once the focus of the research is on teachers using the technology in the classroom (Phase 3) . In contrast, some of the protocols should be used during multiple implementation phases, initially to establish a baseline, and then to track developments and changes over time . For example, researchers should make sure to interview school leaders and teachers at multiple stages of the implementation . This toolkit contains a single version of each research instrument. However, instruments that can be used across different implementation phases contain suggestions for adapting the protocol depending on when you are using it .

Design O

verview

Table 2. Research Instruments by Technology Integration Phase

ETR PHASE 1 ETR PHASE 2 ETR PHASE 3

Integration Context Form

Integration History Interviews

School Leader Interviews

IT Coordinator Interviews

School Tour Checklist

Community Learning Center Staff Interviews

Teacher Interviews

Classroom Observations

Student Interviews

Home Use Interviews

level . Because school or center leaders have contact with teachers, staff, students, and families as well as officials at local and regional education authorities, their comments may reveal incongruence, if any, between local needs, goals, and capacities and national or regional policy .

It is also likely that school or center leaders will be in a position to compare the Intel education technology integration effort with previous reform and development efforts. At the end of their interview, they will be asked to identify candidate teachers for partici-pation in interviews . Any background information they provide on recommended teachers will help to contextualize the observations and interviews researchers will be carrying out within the broader school setting .

11


Interviews with Information Technology CoordinatorsFor school and community center settings (school, local educational authorities, community learning centers, etc .) that have technology coordinators, the Information Technology Coordinator Interview Protocol supports the collection of descriptive information about technology use in the setting before and after the Intel technology integration, the challenges encountered with the device and the integration, the technical support available to information technology professionals and the nature of support that information technology coordina-tors provide to teachers and staff. Their perspective will contribute to the understanding of the degree to which technology use is integrated with the overall goals of the organization and the level of support that is needed and provided for technology use .

School Tour ChecklistUsing the School Tour Checklist, researchers can gather information about the school environment, with a particular focus on the technology infrastructure that exists at the school . The Checklist provides an opportunity to capture information about the school setting, such as size, condition, cleanliness, decorations, etc . It also provides a structure for capturing information about the technology that is available at the school (including quantity, types, and distribution across classrooms and computer labs) and related factors such as cost . Because of the emphasis on technology infrastruc-ture, it may be useful to conduct the school tour with the IT Coordinator or other staff person knowledgeable about technology at the school .

Interviews with Teachers or StaffThe Teacher Interview Protocol supports the collection of data in school settings that characterize technology use, situates it within the context of the overall classroom practices, and builds understanding of how technology use and classroom practices have changed during the Intel integration . For out-of-school integra-tions, the protocol designed for community center staff aims to collect similar kinds of information . In these interviews, teachers or staff will be asked broadly about their challenges and interests to determine the extent to which the technology use fits with their overall work needs. Teachers or staff will also be asked about student use of the device and to comment on changes they may have observed or implemented .

Ideally, classroom observations will also take place for the same teachers who are interviewed . In that case, teacher interviews will also prepare researchers for the classroom observations . Researchers will ask teachers about the lesson planned for the observation period, the learning goals of the activities, and its connection with prior and subsequent lessons .

Interviews with StudentsUsing the Student Interview Protocol, researchers will ask students about the nature of their use of the device across settings and probe for how their use of it may have evolved . If possible, interviews will include a component where students demonstrate their normal patterns of device use to researchers . Questions will also focus on the role of parents, other family members, and friends in supporting students with the use of the technology . Researchers will ask students both about their school-related use (as appropriate) as well as other informal uses of the technology .

Classroom ObservationsIn school- or community center-based integrations, researchers will carry out classroom (or activity setting) observa-tions of the same teachers or staff who participated in interviews . Researchers will approach their classroom observa-tions with the goal of obtaining rich descriptions of how the device is being used and how it is integrated with classroom activities . The loosely structured protocol includes narrative as well as other closed-ended items to allow researchers to focus their attention on capturing the events of the classroom in detail while ensuring that essential quantitative information concerning activities and participation are also documented . The same protocol may be used at various stages of data collection .

Home Use InterviewsIn integrations where students take devices home with them at the end of the day, home use interviews will be conducted to learn more about device use in the home . The home use interview will be conducted with participating students and their family members . Teachers or Community Center Coordinators will be asked to recommend participants for the home use interviews . The goal of the home use interview is to learn about device use patterns in the context of the home, with a view to understanding if and how use patterns vary across settings and how the devices may have influenced student learning in the home .

Des

ign

Ove

rvie

w


Step 5: Analysis and Reporting

Qualitative Reporting To report qualitative data, researchers will be provided with the Qualitative Report Template, which is thematically organized to capture interview and observation data across sites and participants . It will be important to highlight findings related to policy, integration successes and challenges, device use, supports, capacity, and change . This information will enable interviewers to begin to make contrasts, also within the template, between new technology use and what was already possible and/or happening; integration design and implementation; intended change and emergent change; official positions and representations with empirical data; and policy-level discourse and local actors’ experiences and understanding of phenomena .

Triangulation of Findings As with all qualitative research, integrating research findings across instruments and data sources will be necessary to strengthen the validity of individual claims . The Data Triangula-tion Matrices are designed to help structure this process and to identify the convergences and divergences in the information gathered from different data sources on key themes . The triangulation process is intended to assist researchers to construct with rigor and validity a case study that tells the story of implementation .

Synthesizing Findings Another important part of the research process will be to synthesize key findings from the data. To this end, researchers are asked to consider questions such as those below to help them make recommendations from lessons learned, success factors, and course corrections to shape future integrations; locate findings in context; link observed changes to implementa-tion characteristics; and characterize the degree to which changes are systemic and potentially transformative .

CROSS-INTEGRATION COMPARISONS

If budget, research capacity, and data

are available, researchers can use the

structured formats of the reporting

templates to explore comparisons

of the findings from this specific

integration with other previously

studied integrations.

This may be helpful in uncovering

idiosyncrasies of their local integra-

tion that may not be evident given

their in-country perspective. The

comparison process will also be most

helpful in linking practices and poli-

cies with information on successes

and challenges.


DataCollection



DataCollection



DataCollection



DataCollection



DataCollection



DataCollection


Design O

verview


Step 6: Post Analysis After the research on the technology integration effort, the local research team and Intel should meet to review and iterate on the research toolkit . This meeting will be used to review the toolkit’s value and identify recommendations for toolkit modification as needed.


DataCollection



DataCollection



DataCollection



DataCollection



DataCollection



DataCollection


Context: What are some of the most important prior conditions and organiza-tional aspects of the integration that have affected its implementation? In what particular ways did the specific contextual factors affect technology integration efforts? How did the integration strategi-cally address the needs inherent in the context?

Goals: How have the integration’s goals become more refined or shifted during the course of implementation? How did the particular features of the integration affect these goals and vice versa? Were there mismatches or special congruen-cies between the goals and actual implement-ation practices that affected the technology integration?

Use: What are some of the most important or interesting aspects of how the 1:1 technology solutions and other elements of the package have been used in this setting? In school settings, what types of learning or teaching practices has it enabled? In the home or community center, who it using it and in what ways?

Successes and challenges: In what ways was the integration effort most consistently successful? What were some of the more surprising or difficult challenges? Which challenges could have been most readily avoided? Which were insurmountable? What are some of the essential lessons about best practices that were learned from this implementation?

Changes: How have the intended beneficiaries (schools, communities, families, or individuals) been changed by the integration? Are there ways in which they were expected to change but did not? Are there unexpected changes? How are the changes connected to the features of integration? How do changes resulting from integration effort “cluster” (that is, correlate with one another)?

Perceptions: How do stakeholders and beneficiaries feel about the technology integration? Do they consider it a success? Valuable? Are there ways in which they are disappointed? Are there positive outcomes that were unexpected? How do stakeholders and beneficiaries see the integration in ways that differ from a more public perspective?

Other: What other issues or features of the integration do you think are important to consider? For which stakeholders or beneficiaries?

Des

ign

Ove

rvie

w


Design O

verview


Research Step Research Instrument/Resources

ESTABLISHING THE SETTING


Integration History Interview

CUSTOMIZATION Research Customization Guidelines

RECRUITMENT

DATA COLLECTION

School Leader Interview Protocol

Information Technology Coordinator Interview Protocol


Community Learning Center Staff Interview Protocol*

Teacher Interview Protocol

Classroom Observation Protocol

Student Interview Protocol

Home Use Interview Protocol*

ANALYSIS AND REPORTING

Data Triangulation Matrices

Qualitative Report Template

POST-ANALYSIS

Toolkit Summary Table

*Optional activities

Tool

kit S

umm

ary

Tab

le


Instructions The purpose of this form is to gather information to help local researchers better understand the integration context and to inform the customization of the research design and the data collection instruments in the toolkit .

This form asks for basic information about policies related to education, economic development, and ICT infrastructure in the particular integration setting . Focus your attention on the priorities you select in Question 1 . For example, if economic development is a greater focus in this technology integration effort than educational improvement, spend more time on the questions in that section .

In addition, many of the questions below refer to national as well as regional context . Although both are often relevant, please emphasize the setting of the integration while asking the questions. In other words, if the integration effort is a national initiative, focus on the national context; if it is a regional or local initiative, pay attention to the regional or local context .

You may already know most of the information requested in this form . For any information that you do not know, please refer to basic, authentic sources (for example, government web sites) to complete the form . If you refer to any additional resources, please note the source .

Integration Priorities1 . What are the general policy goals of this education technology integration? (Check all that apply.)

• To improve education

• To stimulate economic growth

• To expand ICT access

• Other _______________________

Education Policy2 . What is the basic structure or organization of the national and regional education system?

What is the name of the national education authority? What are the regional and local education authorities that oversee education, and what are their main responsibilities?

3 . What is the basic funding model for the national and regional education system?

Where do schools get their funding? What determines the amount of money that schools receive (for example, do all schools get the same amount, or does it vary by number of children)?

4 . Are students required to take any national standardized assessments?

Are all students in the country tested? If so, in what grades? For what subjects?

5 . What are the most important strategic educational initiatives currently being implemented in the integration setting?

Please briefly list the main activities and goals of each initiative.

6 . Does the setting already have programs focused specifically on educational technology?

Please describe the main activities and goals of any existing educational technology programs in the following areas:

• Access to ICT

• Teaching and learning

• Technical skills

• Distance learning

• Curriculum and assessment

7 . Is there a general plan or policy around expanding ICT use at the national or the regional level?

If so, what are its main components? Who does it impact? What are its goals? How is this national or regional ICT policy or plan related to the ICT policy for education?




8 . Does the integration context have a national or regional ICT policy or plan for education?

If so, what are the main components of the policy or policies? What are the main instructional ICT policies or goals of the national and regional education systems? What are contended or disputed areas of educational policy that might be related to this integration?

9 . To what degree is the education system centralized or decentralized?

To what extent does the national government control education at the regional, local, and school levels? How much autonomy do regions and schools have to make educational decisions such as:

• Allocating funds in the school budget

• Selecting curriculum

• Developing and implementing student assessments and examinations

• Determining the pedagogical approach used in classrooms

• Deciding the scope and direction of educational reform initiatives

10 . What are the certification requirements for teachers?

What level of education and training is required in order to be a teacher in this setting? Does this vary by region? Are the majority of teachers in the country certified?

11 . Do teachers typically use technology in their instruction? In general, how comfortable do teachers feel about using technology in their teaching practice?

12 . What is the status of the education system and services in each of the following areas? Please also describe the extent to which quality varies across different regions.

• Facilities (Are the schools mostly in good condition?)

• Equipment and resources (What are the basic things necessary for schools to operate effectively? Do most schools have the equipment and resources they need?)

• Technology infrastructure (Is it typical for schools to have computers and Internet access? Are there computers in the classroom? Or do students have access to a computer lab? Is technical support, like an IT administrator, available to schools and teachers?)

13 . What is the school calendar year?

When does the school year begin and end? When are the major vacation periods?

14 . How many days of the week are schools typically in session and how many hours per day?

Economic Development 15 . Does the integration setting have a national or regional plan for economic development?

If so, what are the main components of the policy or policies? To what extent is the policy contentious or debated?

16 . To what extent does ICT development relate to the national or regional development plan?

What areas of the development policy might be related to this integration?

17 . What is the basic funding model for national and regional economic development?

How do the national and local governments plan to fund economic growth? What determines the amount of support that various stakeholders receive?

18 . To what degree is the economic development plan centralized or decentralized?

To what extent does the national government control the planning of development initiatives at the regional and local levels? How much autonomy do regions and local businesses have to make economic decisions?

19 . To what degree does the national economic development plan align with other policy priorities selected in Question 1?

Inte

grat

ion

Con

text

For

m


ICT Policy20 . Does the integration setting have a national or regional plan for expanding access to ICT and/or developing

ICT infrastructure?

If so, what are the main components of the policy or policies? Which government entities are in charge of developing or carrying out the policy? What areas of the ICT plan might be related to this integration?

21 . What is the basic funding model for the ICT plan?

What determines the amount of money dedicated to the plan? How is the money allocated?

22 . To what degree is the ICT plan centralized or decentralized?

To what extent does the national government control ICT policy at the regional and local levels? How much autonomy do regions and local authorities have to make decisions such as:

• Allocating funds in the plan

• Selecting the technology to distribute

• Monitoring progress

• Deciding the scope and direction of digital inclusion initiatives

23 . To what degree does the ICT policy align with other policy priorities selected in Question 1?

Prior Technology Integration Efforts24 . Please describe any prior technology integration efforts that have taken place in this geographical area.

25 . Was research or evaluation conducted on these technology efforts? Was any of this research or evaluation conducted by third party researchers?

26 . If prior technology integration efforts have been evaluated, what were the main findings and conclusions?

27 . Are reports and resources from prior rollouts available to decision makers? To what extent are the findings informing the vision and planning for the current integration?

Other Relevant Policies and Plans28 . Please describe any other policies and plans that may provide relevant context for understanding the integration.

Other Information29 . Please describe any other important information about the integration setting.



InstructionsThe purpose of this interview is to gather information from implementation stakeholders about the integration, specifically the role of the technology-based solution in the larger landscape of educational change, economic development, and digital inclusion in the integration setting . This interview should take approximately 1 hour .

For each part of the interview, begin with the initial open-ended question . The topics listed on the left-hand column are intended to guide your discussion . For each topic, the sample probes listed on the right are intended to initiate conversation and elicit information . Please note that the probes included here are suggestions to help you obtain information about the topic . You are not required to ask every question listed; you are also welcome to make up other probes as might be appropriate and relevant . However, we have marked some probes with two asterisks (**) to indicate their high level of importance . Probes with one asterisk (*) are also important, but less so .

Questions

Part 1: Rapport Building and Background Information

Please tell us a little about yourself .

Background

1 . Please tell us about yourself and your role .

2 . How long have you been in this position?

3 . What are some key focus areas for you? What are some goals you are working toward in these focus areas?

Part 2: Policy Context

Please tell us a little about the larger context in which the technology-based solution is/will be deployed . We are really interested in understanding how the integration fits into other initiatives for educational change, development, and inclusion .

General Policy Context

4 . Tell us about the setting in which the technology-based solution is being/will be deployed .**5 . What are the strategic educational, social, or economic initiatives of current importance in this context?

6 . Are there contended or disputed areas of policy that are relevant for this integration effort?

ICT Policy

7 . What is the role of technology in important educational reform or economic development initiatives?**

8 . Is there an ICT policy or plan that is relevant for the integration setting? What are its main components? What is the basic funding model for the ICT plan?

9 . Is there a plan for economic development that is relevant for the integration setting?

10 . To what extent does the plan for economic development relate to the ICT policy for education?

Integration History Interview Protocol

Inte

grat

ion

His

tory

In

terv

iew

Pro

toco

l


Part 2: Policy Context—continued

Overall Goal of the Project

11 . What is the goal of the broad project or initiative in which the technology integration is located?

12 . What is the role of the technology solution in this project or initiative?

13 . What kinds of changes are you anticipating for the broad ICT policy or project? In relation to that, what kinds of changes do you expect the technology solution to trigger?**

Part 3: Technology Integration Goals and Planning

Please tell us a little about the integration. What goals does it serve? Who is it intended to benefit?

Background

14 . Who is the primary sponsor of the education technology integration effort? Who are the secondary sponsors, if any?

15 . How is this technology integration effort being financed?**

16 . How was the initiative conceived? What is the vision behind the initiative?**17 . Who were the key champions? Who are the other key stakeholders who were less involved? (Probes

for: the specific interests and contributions of the stakeholders; the processes in place to foster communi-cation among stakeholders?)**

18 . What are the goals of the integration? What is it intended to accomplish?**19 . To what extent is this integration effort informed by research on prior rollouts? Are reports (especially of

third party research) and resources available?

Key Characteristics

20 . Can you briefly describe some of the key characteristics of the technology integration? (Probe for: Whether the rollout focused on schools, after- or out-of-school settings, or homes and families; if the device rolled out in schools, how many shifts was it used in? Were children allowed to take the devices home?**

21 . What is the size of the integration in terms of its various components? (Probe for: The number of devices that were rolled out; The number of students who used the device)?

22 . What is its cost, and how is the cost being determined?

23 . What devices are being deployed? Have the devices been customized in any way for this context?

Target Audience and Goals

24 . In what settings will the technology solution be deployed?**25 . Who is the integration designed to benefit (schools and teachers, families, etc.)?**26 . What is the target age level? Is there a target subject area?

27 . Why was this age level/setting/subject area selected for the integration?

Part 4: Implementation, Supports, and Resources

Please tell us a little about the supports and resources that are part of the integration . We’re interested in understanding how implementers will be trained in using the devices .

Activities

28 . What are the components of the technology integration package?

29 . What was the process for determining the components of the integration package?

30 . What activities or supports does the integration offer its participants? (Probe specifically about training or professional development for teachers; training or support for IT administrators; ongoing supports for teachers such as mentoring and instructional coaching; supports for students, families, community members; resources; access to a network of other schools and teachers or community centers and staff .)**

31 . What kind of digital content or curricular material is being developed for use with the technology being deployed? How are these materials (intended to be) used in teaching and learning activities?**

32 . What kind of infrastructure preparation does the technology integration require? How is that being handled?**33 . Who is responsible for implementing the activities or supports? Who provides oversight for the technol-

ogy integration effort during implementation?**

Public Opinion

34 . How do implementation stakeholders (government officials, university researchers, etc .) refer to the integration?

35 . How do other stakeholders and the general public (including, if appropriate, the media) refer to the integration?

36 . How is the integration positioned in the media? What kind of coverage has it received? What is the overall public sentiment about the integration?

37 . Are there any materials, media articles, or research literature related to the integration that would help contextualize it or give us insight into how it’s positioned publically or politically?

Integration History

Interview P

rotocol


Part 5: Challenges and Critical Success Factors

Please tell us a little about the supports and resources that are part of the integration . We’re interested in understanding how implementers will be trained in using the devices .

Evaluation

38 . How will the program be evaluated overall? (Probe specifically about ongoing formative research, de-signed to inform course corrections, and summative evaluations .)*

39 . How will you assess specific program outcomes?*

40 . What processes, if any, are in place for communicating course corrections to the relevant stakeholders?*

Success Factors41 . In your opinion, what would be critical success factors of 1:1 technology integrations? How will they con-

tribute to the success of this specific integration?

42 . Do you feel that you have the tools and resources to address these critical success factors .

Challenges

43 . What kinds of challenges, if any, have you faced in implementing a technology policy?

44 . What kinds of challenges do you anticipate/are experiencing for this integration effort?*45 . How would you overcome those challenges?

Part 6: Wrap-Up

Finally, is there other important information about the integration that you’d like to share?

Thank you so much for participating! If we have follow-up questions or a follow-up interview in a couple of months or so, would you be willing to participate again? If yes, how could we contact you then?

Follow-Up

If the integration research design calls for multiple rounds of data collection, this protocol can also be used for the follow-up interview . In these cases, it will be appropriate to focus on observed change since the original research took place. More specific guidelines for custom-ization are as follows:

Section Suggested Modification

Part 1 Omit .

Part 2 Focus all questions on changes in the policy context .

Part 3 Omit .

Part 4 Omit questions 27 and 28 . Use other questions but focus on capturing how things have changed since the previous round of data collection .

Part 5 Highlight this section . Change all questions to focus on observed, rather than anticipated, success factors and challenges .

Inte

grat

ion

His

tory

In

terv

iew

Pro

toco

l


Please use your judgment to determine if all topic categories are applicable to this integration; you are also welcome to create other topic headings as might be appropriate and relevant.

• Introduction and Summary (Including a brief preview of the main findings/conclusions)

• Policy Context

• Prior technology integration efforts

• Technology integration goals and planning

• Implementation, supports, and resources

• Outcomes to date

• Research and evaluation efforts

• Challenges and critical success factors

• Recommendations

• Appendices

1 . List of interviewees

2 . A 1-page summary of all interviews in English

3 . List of abbreviations/Glossary

4 . Full bibliography (including URLs for online sources)

InstructionsThe purpose of the Program Report is to synthesize information from content analyses and background research, as documented in the Integration Context Form, and from interview data collection . Please use the topic categories in this outline to structure the report . This outline follows the sequence of topics in the Integration History Interview Protocol .

Program Report Outline

Program

Rep

ort Outline


Why Customize?The Standard Research Design and Toolkit has been developed to support a broad range of integration types and research efforts. Although the broad research questions and approaches are the same across research contexts, our goal is to ensure that the overall research design and instruments serve the specific needs of local researchers studying the integration of technology solutions in a wide range of settings, and their varying research goals .

What Is Involved?The Research Customization Guidelines are intended to support local research teams in making customization decisions that help calibrate the research design, data collection activities, and instruments to the local contexts and research goals before going into the field. We have identified six dimensions of local integration

research that inform the customization process . We assume that researchers come to this task with knowledge about their integration and their research goals . In the customization step, researchers will situate their integration research project within the Standard Research Design and Toolkit and select research components, instruments, and items as appropriate for their needs and goals .

Customization Steps 1 . As you read the dimensions of cus-

tomization listed in the next section, select the protocols that are most appropriate for your research context . For these protocols, note the revisions you decide are necessary for each instrument or resource you will use in your research . The table below can be used to structure your notes .

Research Customization Guidelines

Data Collection Instrument/Resources Describe Customization Planned Here

Research Design

Integration History Interview Protocol

School Leader Interview Protocol

IT Coordinator Interview Protocol


Community Learning Center Staff Protocol




Home Use Interview Protocol

Res

earc

h C

usto

miz

atio

n

Gui

del

ines


2 . Review the protocols and make the revisions you noted above . Add probing questions as appropriate according to the needs identified above . Wherever possible, insert additional questions as probes rather than modifying the topics in the existing interview protocol . This allows for data that include both the locally relevant specifics and responses to questions that are comparable across integration efforts in different contexts .

3 . Substitute general language in the instruments for precise terminol-ogy used in your context . Translate documents if required .

Dimensions of CustomizationFor your research, consider which of these dimensions is central to the integration effort or reform program designers and which are essential in the community . These considerations may influence your decisions about items to omit from protocols or additional probing questions to consider . Integration efforts may be designed to address goals not listed here . These should also be noted as part of the customization process .

4 . Goals of the local integration effort

Each integration effort will be organized to meet different sets of in-stitutional and local goals . Generally, these kinds of devices are adopted on a large scale in hopes of improving one or more of the following education system quality dimensions:

• Educational transformation

• Enhancing access to ICT

• Economic development

5 . Integration research sponsorship and goals

The Standard Research Design and Toolkit can be used in integration research projects with somewhat different emphases, goals, and budgetary constraints . An under-standing of how these factors shape research priorities is useful in selecting research tasks, editing protocols, and informing real-time decisions during data collection .

Consider the research questions below as you customize to meet research goals .

• How is the technology being used in and across settings?

• What supports and contextual fac-tors shape the rollout?

• What can we say about the value of this particular technology integra-tion effort in this setting?

• What best practices are evident in the field and at the policy level?

• What course corrections might address the greatest challenges for this integration?

• How do the design characteristics of the device enable or constrain productive use?

6 . Length and timing of integration research

Some integration research projects will focus on new technology in-tegration efforts, capturing issues associated with start-up, whereas others will study mature programs,

examining how patterns of use evolve across time . Consider whether the focus of your project includes start-up as well as evolution of usage and change or whether one point in time should be used for data collection . This decision will be based on the goals of the research sponsors, the characteristics of the local rollout, and the research budget . Once you have decided on the timing and number of rounds of data collection required, revising the protocols may be necessary to suit the needs and the timing of your data collection . For example, background information can be eliminated in repeat interviews with the same participants, while asking participants to reflect on prior responses may be included .

7 . Local opportunities

Most integration efforts will have something new to offer researchers meriting special attention for improving our understanding of 1:1 technology initiatives . If integration in your setting is in some way the first of its kind or distinct in the field (for example, serving a new age group or distributed with a new reform package), there may be an interest in highlighting this aspect of the integration in the research questions and data collection . This decision should be made with the support of research sponsor(s) . In cases where special local circumstances shape the overall research goals, researchers should consider the addition of probing questions within the existing protocols .

Research C

ustomization

Guid

elines


8 . Institutional context and research participants

Often, technology solutions will be deployed in school contexts, but there will also be instances where the primary device distribution and support take place in out-of-school settings, other institutional environ-ments like community centers, or outside institutional settings entirely . In these cases, protocols designed for school technology leaders and teachers will need to be modified .

When teachers are not present in the rollout context, teacher interviews can be substituted for interviews with the professionals who spend the most time in direct contact with the children or youth . To maintain the nested character of the sampling, children and youth should be recruited from within each partici-pating professional’s daily practice . Leaders and technology coordinators should also be identified from within the same institutional setting .

9 . Role of the device in the broader education context

The integration research toolkit was designed to investigate the factors associated with the success of 1:1 technology rollouts within their context . In some cases, the entire rollout will be a program of distrib-uting devices; in others, the devices will be part of larger education reform initiative, including additional elements such as teacher profes-sional development, improved infrastructure such as Internet access programs, and parent services .

When device distribution is a part of a larger education reform effort, it is important that the integra-tion research investigate the entire reform effort, with special attention to the role of the device within the reform . It is not sufficient to focus data collection efforts on the device itself without building an under-standing of the reform context or the goals for educational transformation . In these instances, the language of the protocols may need to be revised with names and references to other components of the local reform initiatives to remind research partic-ipants of the scope of the research and to ensure that data informs our understanding of the role of the technology solution within the broader educational reform context .

Res

earc

h C

usto

miz

atio

n G

uid

elin

es


InstructionsThe purpose of this interview is to gather a school-level perspective on the implementation of the technology-based solution . This interview should take approximately 45 minutes .

For each part of the interview, begin with the initial open-ended question . The topics listed on the left-hand column are intended to guide your discussion . For each topic, the sample probes listed on the right are intended to initiate conversation and elicit information . Please note that the probes included here are suggestions to help you obtain information about the topic . You are not required to ask every question listed; you are also welcome to make up other probes as might be appropriate and relevant .

Questions


Please tell us a little about yourself and your role at the school .

Professional Background

1 . How long have you been in this role at this school?

2 . Have you been a school leader in other schools?

3 . How long have you been doing this work?

Part 2: School Reform Context

Tell us about the goals for reform and improvement at this school .

School Improvement Goals

4 . Are there national or regional goals for school improvement that this school is working toward? If so, what are they?

5 . As a school leader, what is your vision for improvement?

6 . Where do you see your school in 5 years? What improvements do you expect to see at this school?

Strategies for Educational Transformation

7 . What educational initiatives is your school a part of?

8 . What school-level strategies are you pursuing to meet these goals?

9 . What will you need to do to achieve your vision? What goals have you set for yourself?

School Leader Interview Protocol3

School Lead

er Interview

Protocol


Part 3: Intel Technology-Based Solution

Tell us about how the Intel-based technology solution relates to the goals for reform and improvement at this school .

Integration Description

10 . Describe the technology solution at your school .

11 . Have any policies or practices been put in place to support the successful use of the device?

12 . Were the devices distributed along with professional development for teachers?

13 . Were there any special programs for others at the school?

14 . Which organizations and individuals contributed resources to support the use of this technology?

Expectations for the Technology Integration Effort

15 . What will a successful implementation look like?

16 . How will you measure progress toward that success?

Relationship Between Technology Integration and Vision for School Reform

17 . How does this technology solution fit in with the school’s strategies and vision?

18 . How does it fit with the larger reform program that the school is a part of?

Part 4: Planning and Communication

Tell us about how you planned for the implementation of the technology solution and communicated your implementation plan .

Planning

19 . How did you plan for the implementation of the solution at your school?

20 . What changes did you have to make to existing systems, structures, and processes?

21 . What additional resources did you have to put in place?

Communication22 . What were your strategies for communicating the implementation to teachers?

23 . What were your strategies for communicating to parents and students?

Part 5: Supports

What supports are available to facilitate implementation of the technology solution?

Support for School 24 . What support do you have as a school leader to implement the solution?

Support from School to Teachers

25 . Describe the type of training and support offered to teachers and students at your school specifically to support the integration .

26 . What supports do teachers have to help them improve their ICT proficiency?

27 . What supports do teachers have to help them integrate the technology solution into their instruction?

28 . What supports are available to teachers to troubleshoot or solve technical problems?

Support Among Teachers (through Teacher Collaboration)

29 . How do teachers collaborate to support one another in implementing the technology solution?

Part 6: Usage, Buy-In, and Results

How are teachers and students using the device and with what results?

Buy-In

30 . Who has chosen to participate in the program?

31 . What is the general perception among students and staff about the technology integration?

32 . How have teachers responded so far?

33 . How have students responded so far?

34 . How have parents reacted?

Usage

35 . How do teachers use the device at this school?

36 . For what activities do you see teachers and students using the device?

37 . How have teachers integrated the device with their overall learning goals?

38 . What can you tell me about individual student use outside class?

Sch

ool L

ead

er

Inte

rvie

w P

roto

col


Part 6: Usage, Buy-In, and Results—continued

Results

39 . In implementing this technology solution at your school, what kinds of challenges have you encountered?

40 . How have you addressed these challenges so far?

41 . What do you value about the Intel technology solution? How do you think it can benefit students, teachers, and the school in general?

42 . What additional resources or supports does the school need to take full advantage of this program?

43 . What advice would you give to other school leaders integrating a similar device into their school?

Part 7: Wrap-Up

Finally, is there anything else you would like to tell me about your experience with this program?

Thank you so much for participating! If we have follow-up questions or a follow-up interview in a couple of months or so, would you be willing to participate again? If yes, how could we contact you then?

At the conclusion of the interview, ask the school leader to identify candidate teachers for interviews .

Follow-Up

If the integration research design calls for multiple rounds of data collection, this protocol can also be used for the follow-up interview . In these cases, it will be appropriate to focus on observed change since the original research took place. More specific guidelines for customization are as follows:


Part 1 Omit .

Part 2 Questions may be used without modification .

Part 3 Questions 10, 17, and 18 may be used without modification . Modify questions 11–16 to focus on how the technology solution has evolved .

Part 4 Questions 20 and 21 may be used without modification . Modify questions 19, 22, and 23 to focus on how communication and planning has evolved .


Part 6 Highlight this section . Questions 35–43 may be used without modification . Modify questions 30–34 to focus on how buy-in among stakeholders has changed .


School Lead

er Interview

Protocol


InstructionsThe purpose of this interview is to gather information from IT coordinators about the planning, implementation, and support of the Intel technology solution . This interview will last approximately 45 minutes to an hour .


Questions


Please start out by telling us about yourself and your job .

Job Title and Professional Background

1 . What is your title?

2 . How long have you been doing this work?

Job Duties3 . Please tell us what your job here is like .

4 . Can you describe a typical day for you in this job?

Job Goals5 . What are your goals in this position?

6 . What do others consider to be a job well done for this position?

Part 2: Goals and Purposes of the Program

Please tell me about the Intel technology solution program and your role in it .

The Context of Technology-Supported Reform and Development

7 . Is there an education reform or economic development initiative that the technology solution is a part of?

8 . How does this program fit in with the larger [name of local initiative]?

Information Technology Coordinator Interview Protocol

Info

rmat

ion

Tech

nolo

gy

Coo

rdin

ator

Inte

rvie

w P

roto

col


Part 2: Goals and Purposes of the Program—continued

IT Coordinator’s Responsibilities in the Program

9 . What are your responsibilities in this larger ICT plan?

10 . Are you involved in

a. ICT planning and policy?b. ICT readiness/infrastructure?c. Program implementation?d. Technical training and support?e. School-wide or community-wide resources (for project, schools, teachers, students)?f. Others?How?

ICT Access and Use Prior to the Integration Effort

11 . Prior to the program, what kinds of technology did students have access to?

12 . Prior to the program, what were the most common types of technology used here?

13 . Prior to the technology solution, how did your school or institution manage teacher/student use of ICT (e.g., computer on cart/wheels, block scheduling, etc.)?

14 . How has this changed with the program?

Part 3: School and Classroom ICT Readiness

Please tell me about the ICT infrastructure at your school or center—the equipment, connectivity, space resources, and anything else that supports ICT use .

Connectivity at the School 15 . Is the connection fast and reliably available?

16 . Is there a protocol for Internet security? How are students protected online?

Local Network Infrastructure

17 . Is there a local infrastructure allowing for shared file storage and collaboration between students?

18 . Can students print and scan within the school or center network?

Maintenance and Support

19 . Who carries out maintenance and support tasks for ICT at the school or center?

20 . Are maintenance and support needs being met?

21 . How did the effort to integrate the technology solution influence maintenance and support needs and activities at the school or center?

Part 4: Specifics About the Integration Effort

Please tell me about the Intel devices and their introduction at this school .

Device Description22 . Was the device distributed with included software or other resources?

23 . What are the device capabilities that are most important for this integration effort?

Supports

24 . Have any policies or practices been put in place to support the successful use of the device?

25 . Did teachers or center staff initially receive training? Did others?

26 . Which organizations and individuals contributed resources to support the use of this technology?

27 . What additional resources are needed (if any) to make the program a success?

Part 5: Usage

Please describe the kinds of device uses that are typical with the students and classes at this school or activities at this center .

User Identification28 . Who are the users of the devices at this school or center?

29 . To whom were the devices distributed? How were they selected?

Usage

30 . How have teachers or staff integrated the devices with their instructional practices?

31 . For what activities do you see instructors and students using the devices?

32 . Do you think the devices are used primarily in school or out of school?

33 . How do you support these activities?

Difficulties

34 . What are common problems for users and classrooms/groups?

35 . Who solves problems and how?

36 . What happens if an individual has difficulty outside school or center use?

Information Technology

Coord

inator Interview

Protocol


Part 6 : Level of Buy-In and Value of ICT Integration

Do you have a sense of how much the school community is supportive of the technology initiative?

Engagement from School or Center Leadership

37 . Are there school or center leaders who have been especially engaged in the integration?

Teacher or Staff Buy-In and Engagement

38 . Is there a lot of variety in terms of teacher or staff engagement with the technology solution?

39 . In general, are teachers and staff supportive of and engage, in the technology solution?

40 . Are there teachers or staff who are less interested? Do you know their reasons?

Student Engagement 41 . Have most students reacted positively to the technology solution?

42 . If some students have reacted negatively, do you know what they don’t like about it?

Part 7: Wrap-Up


Thank you so much for participating! If we have some follow-up questions or a follow-up interview in a couple of months or so, would you be willing to participate again? If yes, how could we contact you then?

At the conclusion of the interview, ask the IT coordinator to identify other candidates for interviews .

Follow-Up



Part 1 Omit .

Part 2 Omit questions 7 and 8 .

Modify questions 9 and 10 to focus on how the IT role has changed . Questions 11–14 may be used without much modification, but change “prior to” to “earlier in” in questions 11–13.

Part 3 Focus all questions on changes in infrastructure since last visit .

Part 4Omit questions 22 and 25 .

Questions 23, 24, 26, and 27 may be used without modification .

Part 5 Focus all questions on changes in buy-in and engagement .

Part 6Highlight this section . Use questions 35–43 without much modification .

Modify questions 30–34 to focus on how buy-in among stakeholders has changed .

Info

rmat

ion

Tech

nolo

gy

Coo

rdin

ator

Inte

rvie

w P

roto

col


School Tour C

hecklist


School Profile Today’s date: __________________________________________

School name: __________________________________________

City/neighborhood: _____________________________________

Type of school: ________________________________________

Grade levels: ___________________________________________

Director name: _________________________________________

Number of students: ____________________________________

Number of male students: ______; female students ________

Number of teachers: ____________________________________

Number of male teachers_____; female teachers ___________

Average class size: ______________________________________

Number of classrooms: _________________________________

ICT Infrastructure 1 . Total number of computers in the building (including desktops, laptops):

2 . Average number of computers per classroom (including desktops, laptops):

3 . Number of computer labs:

4 . Average number of computers per lab:

5 . Are there servers that have curriculum content and other software that connect to the computers?

6 . List any other functioning computers (desktops or laptops) at the school (e.g., in the school office/principal’s room/ teachers’ room, etc .):

7 . Total number of whiteboards in the school:

8 . List any other technology devices, and how many of each exists in the school (e .g ., projectors, TV’s, DVD players, etc .):

9 . What is the standard ICT set up in the classroom?

10 . What percentage of classrooms has technology available?

11 . Does the school have Internet access? Wireless?

12 . Does each classroom have Internet access?

13 . What is the monthly Internet cost for the school?

14 . How many local access points via routers in the school, if any?

15 . What is the electricity supply and access in the classroom?

16 . How many wall outlets in each classroom for charging the equipment?

17 . What is the monthly cost for power in the schools?


Other Educational Resources18 . Is there a library or media center?

• How many books and resources?

19 . Do classrooms have classroom libraries?

20 . What other resources do you see available for student or teacher use?

21 . Describe other additional rooms (i .e ., gymnasium, stage, resources room, pedagogical room, etc .) .

School Environment22 . Describe the hallways between classrooms .

• What is on the walls (e.g., student artwork or projects, school vision/mission, etc.)?

• Are the hallways clean and free of litter?

23 . Describe any common space that is available for teachers and/or students .

• Is the space clean and free of litter?

24 . What do students do during break/recess or other unstructured time?

25 . What do students carry with them while at school (e.g., backpack, books, technology)?

26 . Are students and teachers interacting in the hallways? If yes, describe the nature of those interactions.

27 . Is the school leader (e.g., principal) visible and around school?

Sch

ool T

our

Che

ckli

st


InstructionsThe purpose of this interview is to gather information from the coordinator of a community learning center (or other informal learning contexts) on the implementation of the technology-based solution . This interview should take approximately 1 hour .


Questions


Please tell us a little about yourself, your role in this center, and your experience with technology .

Current Experience

1 . How long have you been working at this center? Have you had your current position the entire time?

2 . How long have you been working in these kinds of centers?

3 . Have you ever worked in schools (primary, secondary, college, or university)?

Background with Technology

4 . What experience do you have supporting IT or children’s use of technology?

5 . Tell us about how you use technology in other settings—at home or other workplaces, for example .

Part 2: Learning Context

Please tell us a little about this center and your work at it .

Context

6 . Tell us about the center. What is its primary function?

7 . What kinds of activities are offered? Who are the typical participants?

8 . Can you describe a typical day for participants?

Practice/Activities

9 . What kinds of activities do you coordinate or lead?

10 . What resources do you use to help planning activities?

11 . How do you gauge progress among the participants/learners?

Community Learning Center Staff Interview Protocol

Com

munity Learning C

enter Staff

Interview P

rotocol


Part 3: Intel Technology Solution

Please tell us a little about your experience with using the Intel educational technology .

Example of One Specific Activity

12 . Please describe one activity in which you have used the technology .

13 . How did you plan and prepare for this activity? What resources did you use?

14 . What did you do? What did the children do?

15 . How did the children respond to the activity? Did you assess what they were learning?

16 . What kinds of products, if any, did they produce?

Integration with Daily Practice

17 . What kinds of activities have you been able to use the technology with, in general? (Note: Refer to questions 13-17 for further probes if necessary .)

18 . How have the children responded, in general, to the use of the technology?

Perceived Impact/Value of the Technology

19 . How do you think this program might benefit participants? Children’s learning?

20 . How might it benefit family members or the community beyond participants in this program? Organizations that provide services, businesses, faith-based entities, etc.?

21 . How do you think this program could benefit you as an educator?

Challenges to Implementation

22 . What kinds of challenges and barriers have you encountered with using the technology in the community center context?

23 . How have you addressed these challenges?

Part 4: Professional Development

Please tell me a little about the professional development that you have received as part of the Intel technology solution .

Participation In, and Value of Professional Development

24 . Have you participated in professional development specific to the technology solution and related programs?

25 . If yes, what did your professional development activities consist of?

26 . What did you find most valuable about this professional development?

27 . In what ways do you think the professional development could be improved?

Part 5: Support for Implementation

Please tell us about the various kinds of support you have received to help you implement the technology (support for implementation, technical support, support from colleagues, etc.). Please also tell us about any difficulties you may have encountered and how they were addressed .

Support for Implementation

28 . How was the Intel technology solution explained to you at the start?

29 . What support were you offered to help you learn about start using the technology? Have you received all the support you were offered?

30 . Has the implementation been carried out in a way that reflects the initial plan? If not, what are some of the major differences?

Technical Support

31 . Have you encountered any technical difficulties in using the technology?

32 . To what extent do you receive support to help address these difficulties? (Probes: Who provides the support? How often do you need the support? How often do you receive it? What is the nature of the support you receive? Is the support you receive always in response to a problem or is it also proactive?)

Support from Community Center Leadership

33 . To what extent do you feel supported by the leadership at the center in implementing this technology? And in resolving the challenges you have faced?

Support from Colleagues34 . Do you have colleagues at this center or beyond with whom you can share your experiences?

Have your colleagues shared their experiences with you?

35 . Has interacting with your colleagues helped you use this technology?

Com

mun

ity

Lear

ning

Cen

ter

Staff

Inte

rvie

w P

roto

col


Part 6: Wrap-Up



Follow-Up



Part 1 Omit .

Part 2 Omit questions 6 and 8 .

Modify questions 7 and 9–11 to focus on changes, if any, since the previous visit/interview .

Part 3Questions 12–18 may be used without much modification .

Focus questions 18 23 on changes since the previous interview, modifying the phrasing as appropriate . If possible, refer back to the interviewee’s prior responses while asking these questions .

Part 4 Focus questions 24–27 on changes if any since the previous interview, modifying phrasing as appropriate .

Part 5Omit questions 28–30 .

Focus questions 31–35 on changes since the previous interview, modifying the phrasing as appropriate .

Part 6 May be used without modification .

Com

munity Learning C

enter Staff

Interview P

rotocol


InstructionsThe purpose of this interview is to gather information from teachers about the implementation of the Intel technology-based solution . This interview should take approximately 1 hour .


Questions


Please tell us a little about yourself as a teacher .

Current Experience

1 . What grade levels and subjects do you teach?

2 . How long have you been teaching at this school?

3 . Do you have any other roles at the school (like departmental head or instructional coach) in addition to being a classroom teacher?

Background4 . How long have you been teaching?

5 . How much experience do you have with using technology in the classroom to support your teaching?

Part 2: Classroom and School Context

Please tell us a little about your school and how you teach .

Classroom Practice

6 . Can you describe a typical lesson in your classroom?

7 . What resources do you use while planning your lessons?

8 . How do you gauge your students’ progress?


Teac

her

Inte

rvie

w P

roto

col


Part 2: Classroom and School Context—continued

School Reform Context

9 . What are the expectations that your school administration or higher level administrators have of you?

10 . Has technology changed your job as a teacher? If so, how?

11 . Right now, are you actively trying to change the ways teaching and learning happen in your classroom? What are you doing that is new or different?

12 . Is this a change that others in your school or community are also working on, or are you doing this on your own?

Part 3: Intel Technology Solution

Please tell us a little about your experience with using the Intel educational technology .

Integration with Classroom Practice

13 . For how long have you been using the Intel technology solution in your classroom?

14 . Please describe one lesson or activity in which you have used the technology . (Probes: What was the activity? What did you do? What did students do? What sorts of resources did students use? What kinds of products, if any, did students produce? How did you assess what students were learning?)

15 . How did you plan and prepare for this lesson? What resources did you use?

16 . How would you have taught this topic or this lesson before you had technology?

17 . What kinds of lessons and activities have you been able to use it with, in general?

18 . How do you decide when to incorporate technology into your lessons, or when to do a certain lesson or activity without technology?

19 . How have your students’ responded to the use of the technology?

20 . How comfortable are the students with using the technology? Do they have the skills they need to use the technology productively? Do they have any trouble using the technology?

Perceptions/Buy-In21 . How do you think this program could benefit your students?

22 . How do you think this program could benefit you as a teacher?

Challenges 23 . What kinds of challenges and barriers have you encountered with using the technology in the classroom?

24 . How have you addressed these challenges?

Prior Technology Experience

25 . Did you use technology in your classroom previously? If so, how?

Part 4: Professional Development

Please tell me a little about the professional development that you have received as part of the Intel technology solution .

Professional Development Participation

26 . Have you participated in professional development specific to the technology solution and related programs?

27 . If yes, what did your professional development activities consist of?

28 . What did you find most valuable about this professional development?

29 . In what ways do you think the professional development could be improved?

Part 5: Support

Please tell us about the various kinds of support you have received to help you implement the technology (support for implementation, support from the school administration, technical support, support from colleagues, etc.). Please also tell us about any difficulties you may have encountered and how they were addressed .

Support for Implementation

30 . How was the Intel technology solution explained to you at the start?

31 . What support were you offered to help you learn about and start using the technology in the classroom? Have you received all the support you were offered?

32 . Has the implementation been carried out in a way that reflects the initial plan? If not, what are some of the major differences?

Technical Support

33 . Have you encountered any technical difficulties in using the technology?

34 . To what extent do you receive support to help address these difficulties? (Probes: Who provides the support? How often do you need the support? How often do you receive it? What is the nature of the support you receive? Is the support you receive always in response to a problem or is it also proactive?)

Support from School Administration

35 . To what extent do you feel support from school leadership/administration for the integration of technol-ogy into instruction?

Teacher Interview P

rotocol


Part 6: Wrap-Up



Follow-Up



Part 1 Omit .

Part 2 Omit questions on changes since the previous interview .

Part 3

Questions 12–15 may be used without modification .

Focus questions 16–19 on changes, if any, since the previous interview, modifying phrasing as appropriate .

Omit question 20 .

Part 4 Focus questions on changes since the previous interview .

Part 5Omit questions 25–27 .

Focus questions 28–32 on changes, if any, since the previous visit, modifying the phrasing as appropriate .

Part 6 Wrap up .

Part 5: Support—continued

Support from Colleagues36 . To what extent have you been able to share your experiences with the technology with your colleagues?

Have your colleagues shared their experiences with you?

37 . Has interacting with your colleagues helped prepare you to use technology in class?

Teac

her

Inte

rvie

w P

roto

col


InstructionsYou can also use this protocol to interview a teacher whose class you have just observed . In these cases, it is appropriate to focus on the lesson itself, including planning, activities, student responses and engagement, technology use, etc . Here we provide guidelines and questions for how to use this protocol for discussing the classroom observation .

• Replace Section 3: Intel Technology Solution with the Classroom Observation Debrief section provided below . Here we have adapted the questions from Section 3 to address the particular lesson that was observed .

• Use the other sections of the Interview Protocol as appropriate, depending on whether this is your first interview with the teacher or a follow-up interview .

• If you have already conducted both an initial interview and a follow-up interview with this teacher, you can use just the Classroom Observation Debrief section below to discuss the observed lesson .

Classroom Observation Debrief

Lesson Background

1 . What grade-level are the students in this class?

2 . Are these students generally working at, above, or below grade-level in terms of their achievement and performance?

3 . How did you plan and prepare for this lesson? What resources did you use? Or did you design the lesson/activity yourself?

Today’s Lesson

4 . What were your goals for today’s lesson?

5 . Did you accomplish what you hoped to accomplish? What were the strong points from today’s lesson?

6 . How How did students respond to today’s lesson? How did they respond to the technology component in particular?

7 . How did the technology support student learning during today’s lesson? Please describe.

8 . How did the technology support you in your role as the teacher during today’s lesson? Please describe.

9 . Did you encounter any challenges during today’s lesson? Did you run into any problems with the technology in particular?

10 . Are there any things you would do differently if you were to teach this lesson again? Please describe.

11 . How would you have taught this topic or this lesson before you had technology?


Classroom

O

bservation P

rotocol


Part 1: Background Information

OBSERVER(S):

OBSERVATION DATE: (DD/MM/YYYY) _ _/_ _ /_ _ _ _ OBSERVATION START TIME:

LENGTH OF THE OBSERVATION (MINUTES): OBSERVATION END TIME:

SCHOOL NAME: DISTRICT / LOCAL AUTHORITY / REGION:

TEACHER NAME: SUBJECT:

NUMBER OF STUDENTS: NUMBER OF BOYS: NUMBER OF GIRLS: TOTAL:

AVERAGE STUDENT AGE:

Teacher’s stated goals for the lesson: (If possible, speak with the teacher before the observation begins and complete this section with the following information: What is the teacher planning to do? How does the lesson/activity fit in with the unit that the class has been doing before? Are there particular outcomes the teacher is hoping for?)

Physical Arrangement: (Draw or describe the physical arrangement of the classroom .)

Technology: (Describe the technology resources present in the classroom and include the number of each . Fixed technology resources, like desktop computers and projectors, can be included in the diagram of the classroom above .)

Part 2: Observation Notes

In this section, please take detailed notes in real time as you observe classroom activities .

The following questions serve as guidelines for what you will document during the classroom observation . Your descriptions of all the classroom activities should include answers to questions 1–6 . For each topic/question, please note what you observe in the left-hand column; you may use the right-hand column to note your hypotheses and conjectures about what you think .

Structure of the Lesson

Describe the structure of the lesson that you observe. What is happening in the classroom? What are the teacher and the students doing?

What You See What You Think

Cla

ssro

om

Ob

serv

atio

n P

roto

col

NOTE: If you have not interviewed this teacher previously, please schedule a full interview with each teacher independent of the observation (see the Teacher Interview protocol in the Intel Education Transformation Toolkit).


Part 2: Observation Notes —continued

Interactions Between the Teacher and Students

How do the teachers and students interact? Try to capture examples of the type of questions teachers ask students and how students respond, as well as the questions students ask teachers and the teacher’s responses .

In addition to questions, please also note the other ways in which the teacher and the students interact .


Interactions Among Students

Do students have an opportunity to interact with one another? If so, how do they interact? Do they work on a task together? Do they provide feedback to one another? What role, if any, does the technology play in these student interactions?


Use of the Technology/Device

What activity are teachers and/or students doing with the technology? What is the teacher doing with the technology? What are the students doing with the technology? How does the technology influence how students are interacting with the content? How does the technology influence how students are interacting with the teacher and with each other? What kinds of products are the students producing with the technology? Are teachers or students experiencing difficulties in their use of the technology/device? Are they able to troubleshoot?


Use of Other Resources

What other resources does the teacher use? Note the materials that the teacher uses during the lesson (for example, chart paper, blackboard, visual aids, etc.). What other technologies are teachers and/or students using during the lesson?


Other Observations

What else is characteristic of what the teacher does? What else do students do?


Classroom

O

bservation P

rotocol


Part 3a: Reflections on the Lesson Overall

Please reflect on the lesson and complete the following questions as soon as possible after the observation.

1 . What is the teacher’s overall approach to classroom instruction (facilitator, classroom manager in control, teacher as co-learner, etc.)?

2 . Did the students seem to be clear on what they were supposed to do during the lesson, or were they confused about how to do the activity?

3 . What components of the lesson/activity did students seem enthusiastic about? Include specific examples of student comments and actions to illustrate .

4 . Had the teacher prepared any other activities or tasks for students to do in case something went wrong with the technology or students finished early?

5 . What other reflections do you have about the lesson?

Part 3b: Reflections on the Technology Activity


6 . Where was the locus of control for the technology-rich learning activity (or activities) you observed (i.e., students, teacher, software)? Please describe .

7 . Did the technology give users access to instructional resources and/or content information? If yes, what resources and how are they different from what is available without technology?

8 . How are students using the technology to interact with content in new ways (e .g ., creating content, finding video, applying skills or abilities, etc.)?

9 . Did the technology activity allow students to build or practice higher order thinking skills (e .g ., problem solving, reasoning, synthesizing information)? Please describe.

10 . Did the technology enriched learning activities allow students to reflect on themselves as learners (e .g ., through embedded assessments, viewing products, building portfolios)? Please describe.

11 . Did teachers, students, and families use technology to collaborate and/or communicate with each other? Please describe.

12 . Did the technology enable students to engage in individualized or personalized learning and/or to have some autonomy over their own learning? Please describe.

Cla

ssro

om

Ob

serv

atio

n P

roto

col


Part 3b: Reflections on the Technology Activity—continued


13 . How did the students respond to the technology used? (Did they seem bored, interested and involved, etc.?)

14 . Was there anything about the technology that seemed difficult for the teacher or students to do? Did any glitches with the technology impede the progress of the lesson?

Classroom

O

bservation P

rotocol


InstructionsThe purpose of this protocol is to gather stories and examples from students about their expectations and experiences integrating the Intel technology-based solution . For each section, begin with the open-ended question . Use probes as needed to ensure that you obtain all the information listed in the left-hand column . Example probes are provided on the right, but feel free to make up your own . You do not need to ask every sample question listed in the right-hand column . Allow the interview to flow naturally and look for opportunities to ask follow-up questions when interesting topics arise.

If possible:

• Select a diverse group of students .

• Ask students to bring the device with them to the interview .

• Tape record the interview for future reference .

• Introduce yourself, the purpose of your visit, and set expectations for the interview before you begin .

Questions


Let’s go around the room . Please tell me your first name, what grade you are in, and how long you’ve been using the device .

General Rapport Building

1 . What do you like to do with your free time?

2 . What are your favorite subjects in school?

3 . What do you want to do when you grow up?

Part 2: Device Use Demonstration

Can you show me something interesting you’ve learned to do with the device?

Use Demonstration

4 . How did you learn to do that?

5 . How often do you do this in school?

6 . Do you do this with the device at home or elsewhere outside school?


Stud

ent

Inte

rvie

w P

roto

col


Part 3: Device Use Stories

Tell me about a recent time when you were asked to use the device by an adult .

Use Description

7 . What did you do? Why?

8 . Who else was involved?

9 . What did you learn?

Part 4: General Use Patterns

What are some of the ways you frequently use the device?

Use Types

10 . I’m going to read a list of different ways you might use the device . For each item I read, raise your hand if this is something you do frequently in school or your community center . Note: Read the activities listed below one at a time, counting the number of students that raise their hands for each . If you are on sched-ule, ask a few students as they raise their hands to give examples . Depending on how the conversation has been going, this is a good opportunity to even out participation among students . a . Do research for schoolworkb . Type your assignmentsc . Take notes, quizzes, or testsd . Send e-mail to friends or teacherse . Chat with friends or teachersf . Create presentations and projectsg . Collaborate with othersh . Draw pictures or work with photosi . Write stories, blogs, wiki content, fliers, or publicationsj . Play games

Use Frequency and Setting11 . How often do adults ask you or allow you to use the device?

12 . What subject(s) do you mostly use the device for?

Part 5: Reflections

Can you tell me what you think about the device?

Likes and Dislikes

13 . What are some of your favorite things about using the device?

14 . What do you like least about using the device?

15 . If you could make improvements to the device, what would you do?

Impact16 . What’s different now that you are using the device?

17 . If you use computers at home or anywhere else, how is using this device different?

Student

Interview P

rotocol


Part 6: Other Information

Is there anything else you would like to tell us about what you’ve done or would like to do with this device?

Follow-Up



Part 1 Use without modification, particularly if these are new participants .

Part 2 Use without modification .


Part 4 Use question 10 as is, but ask students to describe how their use has changed . Change questions 11 and 12 to focus on changes in use .

Part 5 Use without modification . Highlight questions 16 and 17 .


Stud

ent

Inte

rvie

w P

roto

col


InstructionsThe purpose of this protocol is to gather stories and examples about device use in informal settings—namely, the home . Ideally, this interview should be conducted with multiple family members, including the primary user of the device . For each section, begin with the open-ended question . Use probes as needed to ensure that you obtain all the information listed in the left-hand column . Example probes are provided on the right but feel free to make up your own . You do not need to ask every sample question listed in the right-hand column . Allow the interview to flow naturally and look for opportunities to ask follow-up questions when interesting topics arise .

This interview will take approximately 45 minutes .

Questions


Can you tell me about your family and your home environment?

Family Members

1 . Who am I speaking with?

2 . How many people live in your house?

3 . How long have you been living here?

Occupations4 . What do members of your family do for a living?

5 . How long have they been in their profession(s)?

Part 2: General Technology Experience

What kinds of technology do you frequently use?

General Technology Ownership

6 . Do you own a computer? If so, how many and since when?

7 . Do you have an Internet connection in your home? If so, since when?

8 . If you do not have Internet at home, where and how do you access the Internet?

9 . What other information technology do you own (television, radio, etc.)?

Previous Technology Use10 . Before owning the device, how often did you use these technologies? Where and how did you access

these technologies before owning your own?

11 . Before owning the device, what did you use technology for?

Home Use Interview Protocol

Hom

e Use

Interview P

rotocol


Part 3: Device Usage

How do you use the device in your home?

Device Users

12 . Who is the primary user of the device?

13 . Who else in your home uses the device?

14 . Does anyone outside your home use the device?

Types of Device Use15 . What do people in your home use the device for?

16 . How frequently do you use the device for particular purposes?

17 . I’m going to read a list of different ways you might use the device . For each item I read, tell me if this is something you do and how often . Note: Read the activities listed below one at a time, noting responses for each item . If you are on schedule, ask family members to give examples and provide more detail . Depending on how the conversation has been going, this is a good opportunity to even out participation among family members .a . Find information on the Internetb . Communicate with friends or family (e-mail, chat, or instant messaging)c . Do schoolwork d . Manage financese . Participate in social networking activitiesf . Draw pictures or work with photos g . Write stories, blogs, wiki content, fliers, or publicationsh . Play games

i . Watch videos

18 . Now let’s talk about your children . I’m going to read a similar list of ways that people use the device . For each, tell me is this is something your children do and how often .a . Find information on the Internetb . Communicate with friends or family (e-mail, chat, or instant messaging)c . Do schoolwork d . Use social networking (Facebook, Twitter, etc .)e . Draw pictures or work with photosf . Write stories, blogs, wiki content, fliers, or publications g . Play games

h . Watch videos

19 . Do you have any rules related to your child’s use of technology (for example, when they use technology, how much time they spend, what they use it for, what content they access, etc.)? Please describe.

20 . Is there anything else you would like to tell us about what you’ve done or would like to do with this device?

Motivations for Use

21 . What motivates you to use the device?

22 . What do you like most about the device?

23 . Who encourages you to use the device?

24 . Whom do you encourage to use the device?

Barriers to Use

25 . What are some challenges to using the device?

26 . Have you had any technical problems with the device? If so, how did you resolve them?

27 . What supports would you need to take full advantage of the device?

28 . What do you like least about using the device?

29 . If you could make improvements to the device, what would you do?

Hom

e U

se

Inte

rvie

w P

roto

col


Follow-Up



Part 1 Omit .

Part 2 Omit questions 9 and 10 . Change questions 6–8 to focus on changes in general technology use .

Part 3 Focus all questions on changes in use patterns .

Part 4 Highlight this section . Use without modification .

Hom

e Use

Interview P

rotocol

Part 4: Reflection

What has changed since owning the device in your home?

Perceived Impact

30 . Has having the device at home helped you do or find anything you couldn’t do or find before?

31 . Has having the device at home made any difference for you and your family?

32 . Has having the device at home made any difference for your children specifically? Please describe.

33 . If you use computers at home or anywhere else, how is using this device different?


Please use the two tables below to reflect on how each data source informs your understanding of the ideas below. This document is designed to bring together data from various sources to support later analysis and the emergence of key findings.

Describe what you have learned about each item below according to each data source . If detailed descriptions of the topics listed on the left are needed, consult the reporting template .

Matrix of Context-Level DataThis matrix will help bring together data from different sources concerning the integration setting and characteristics.

Interviews with Implementation Stakeholders

Interviews with School or Community Leaders

Interviews with IT Coordinators Other Sources

Context of Integration

Target Population

e .g ., Program is intended for children ages 5–12 . Also serves low-income children ages 12–17 .

Education System Structure and Characteristics

Education Reform Goals

Program Supports

Political Climate

Rollout Characteristics

Rollout Features

Planning

Relationship to Planners of the Integration Effort

Data Triangulation Matrices

Dat

a Tr

iang

ulat

ion

Mat

rice

s


Data Triangulation M

atrices

Matrix of School- and Classroom-Level Data*The following matrix will help bring together data from different sources concerning the technology integration in school and classroom contexts . In some cases cells may be left blank .

Interviews with School Leaders

Interviews with IT Coordinators

Interviews with Teachers


Interviews with Students and Families

Other Sources

School-Level Implementation

Device Access

Supports for Use of Educational Technology

Buy-In and Commitment

Capacity and Readiness

Organizational/ Professional Culture

Professional Autonomy

Teacher Growth

Teacher Collaboration

Leadership (at schools or other learning contexts)

Classroom-Level Implementation

Integration of the Device into Curriculum, Instruction, and Assessment

Typical Classroom Instruction/Activity

School-Level Implementation

Independent Research

Communication

Innovative Instruction

Extension of Learning Beyond the Classroom


Interviews with School Leaders

Interviews with IT Coordinators

Interviews with Teachers


Interviews with Students and Families

Other Sources

Changes in Teaching and Learning

Changes in Teacher Attitudes

Changes in Instructional Practices

Changes in Curriculum and Assessment

Changes in Students’ Learning Behaviors

Changes in Policies and Politics

Unexpected or Emergent Changes

Matrix of School- and Classroom-Level Data*—continued

*This matrix may be adapted, with some modifications, to synthesize data gathered in out-of-school settings as well .

Dat

a Tr

iang

ulat

ion

Mat

rice

s


Please use the headings, questions, and guidelines in this template to write your qualitative report .

1. Study Methods In this section, please describe your sample and provide basic information about (1) the government representatives you interviewed, (2) the sites you visited, (3) people you interviewed, and (4) the lessons you observed . Also, please describe any methods that differed from those described in the research toolkit.

1.1 Stakeholder Sample List the name, title, and role of each of the government representatives interviewed .

1.2 Site Visit Sample Provide a description of each of the schools or community centers that were visited . For each site, include:

1 .2 .1 Name of the site

1 .2 .2 Date of each visit and the stage relative to the rollout timeline

1 .2 .3 A brief description of the site itself, including its location, its physical condition and resources, the economic status of the area, and anything about the surrounding community that would help others better understand the site

1 .2 .4 An overview of the history of reform in the site including a brief description of the level of technology availability and use before and after the integration relative to other sites in the area

1 .2 .5 A description of who was interviewed including site IT administrators, leaders, and an overview of the number of teachers or staff and student research participants

2. Context of IntegrationIn this section, describe what you learned about how features of the large-scale educational or social context for the technology integration (the city, state, or country) have shaped the integration strategy. Focus on factors that might affect the implementation of the integration . Note that school or community context will be discussed below .

2.1 Target PopulationDescribe the target population of the integration effort in terms of its demographics: ethnicity and gender, socioeconomic characteristics, literacy levels, language, and culture . What other features of the target population are relevant to technology integration?

2.2 Education System Structure and CharacteristicsDescribe the overall structure of the education system or relevant aspects of the target community . What are the features of the system that create opportunities or challenges for the integration effort?

2.3 Education Reform GoalsDescribe the overall education goals of the education authority or social agency in charge of the integration effort (local school district, national government, etc.). What are the goals for student outcomes and skills? How is the integration effort expected to support the goals?

2.4 Program SupportsDescribe the supports and resources that the education authority or social agency in charge of technology integration provides to help schools improve . Supports may include professional training, resources, funding, etc . Also describe constraints imposed by the authority that might limit school or community improvement efforts.

2.5 Political ClimateDescribe the political climate surrounding the integration effort. Who are the primary political stakeholders and what do they stand to gain? How is the public discussion of the rollout framed in the media?

Qualitative Report Template

Qualitative R

eport Tem

plate


3. Rollout CharacteristicsIn this section, describe what you learned about the rollout .

3.1 Rollout FeaturesDescribe the basic characteristics of the rollout, including the scope, the characteristics of the devices as deployed, and the broader reform or development context (if any) that the technology integration is situated in .

3.2 PlanningDescribe the planning process for the technology integration effort. What aspects of the context were taken into account? What aspects were overlooked?

3.3 Relationship to Planners of the Integration EffortDescribe the nature of the relationship between the schools or community entities carrying out the technology integration and the planners of the integration effort.

4. Site-Level Implementation In this section, describe various aspects of implementation at the site level across the visited sites . Please include (1) strong or unique examples in each category, (2) what is common or similar across sites, and (3) any important variations or exceptions across sites .

4.1 Device AccessDescribe the integration in terms of technology available in the sites. What is available and where are those resources located? Please describe any other supporting hardware, software, and technology equipment used to support the implementation .

4.2 Supports for Use of the DeviceWhat types of training and supports are available to sites, teachers, staff, or students specifically to support the integration of education technology? What supports do teachers or staff have to help them improve their proficiency with technology? To help them integrate the technology solution into their instruction?

4.3 Teacher or Staff Buy-In and CommitmentDescribe the extent to which teachers or staff support or resist the integration effort. How do they show this support or resistance?

4.4 Teacher or Staff Capacity and ReadinessTo what extent are teachers or staff prepared to implement the integration effort? Do teachers or staff describe specific areas in which they need additional training or support?

4.5 Site Professional CultureDescribe the professional culture and environment at the sites .

4.6 Professional Autonomy How much autonomy did teachers or staff at the sites you visited have with respect to decisions about their day-to-day practices? What was the process of trying new things or learning new things for teachers or staff?

4.7 Teacher or Staff GrowthWas there evidence that teachers or staff were supported in experimentation and professional development? Was professional development directed in part by their interests and goals?

4.8 Teacher or Staff CollaborationDid you encounter evidence that teachers or staff collaborate with one another in their work? Did collaboration happen in connection with the use of the device or other technology? Are there other communities of collaboration in which teachers are involved where they gain ideas, energy, or insight into developing innovative teaching and learning practices?

Qua

lita

tive

Rep

ort T

emp

late


4.9 Site LeadershipDescribe the role of site leaders in relation to the integration effort. For example, are the school leaders you interviewed working to build a common vision of learning within their schools? How do community leaders describe their goals for their communities and youth? Describe the consistency (or lack of consistency) among national or regional goals and school or community leaders’ visions for effective practice.

5. Classroom-Level Implementation (for school-based integration efforts)In this section, describe typical and innovative patterns of technology use (related to the integration) at the schools you visited . Please include (1) strong or unique examples in each category, (2) things that are common or similar across schools, and (3) variations across schools or important exceptions .

5.1 Integration of the Device into Curriculum, Instruction, and Assessment 5 .1 .1 Curriculum

Describe how teachers are incorporating the device into their curriculum. What synergies, if any, are being created between curriculum and the device?

5 .1 .2 Teaching and Learning Describe how ICT has been integrated into the classroom. How has it been used to provide new or improved learning opportunities for students? What challenges related to integrating technology did you hear about in your interviews?

5 .1 .3 Assessment Describe how teachers are incorporating the device into assessment practices. What synergies, if any, are being created between assessment practices and the device?

5 .1 .4 Integration Challenges Did you learn of any device uses that could not be connected with curriculum, instruction, or assessment?

5.2 Typical Classroom InstructionHow do teachers typically make use of the device in their instruction? What do teachers and students do during lessons? Were there any differences between how teachers described their practice and what researchers observed in the classrooms?

5.3 Independent ResearchDid students at the schools you visited use the devices to search for material or analyze information? If so, please comment on this including a description of the diversity of resources accessed .

5.4 CommunicationWere devices used to support the practice and development of communication skills? If so, please comment on how common this kind of use was and on the range of communication styles and formats . Please also note if communication practice ever involved outside audiences .

5.5 Innovative InstructionDescribe any innovative uses of the device for teaching and learning . Examples include involving students in projects, asking students to collaborate, having students assess their own work or the work of peers, etc .

5.6 Extension of Learning Beyond the ClassroomDescribe any efforts to use the device to extend learning outside the classroom. Are students connecting with community members, learning about other cultures, or having the opportunity to learn from experts outside the classroom?

Qualitative R

eport Tem

plate


6. Lesson Descriptions (for school-based integration efforts)Please describe six individual lessons that you observed, providing a paragraph description of each . For your descriptions, please select:

6.1 Three Innovative Lessons To select the most innovative lessons, please consider strong examples of student-centered teaching and learning and powerful or creative uses of the technology integration .

6.2 Three Typical Lessons To select the typical lessons, please consider the type of instruction that is most common in your country (and at the schools you visited) at this point in time .

For each of the descriptions, please describe what happened during the lesson. What were the teacher and students doing? What types of products were students creating? Were students working alone, in groups, or as a whole class? How were the teacher and students interacting?

7. Changes in Community Center Experience (for nonschool integration efforts)In this section, summarize responses from community center staff and student interviews to describe the changes that are taking place as a result of the technology integrations . In each category below, please include (1) strong or unique examples in each category, (2) what is common or similar across sites, and (3) any important variations or exceptions across sites .

7.1 Changes in Staff Member Experience Describe any changes to staff members’ routine practices that are related to the introduction of the device in the site. Include staff members’ own assessment of the importance of these changes or relevance to their overall work goals.

7.2 Supports for Device UseWhat kinds of supports (either technical or substantive) are in place for staff and students in their use of the device?

7.3 Integration of the Device into Daily Practice 7 .3 .1 Student Activities

How have the students’ routine activities changed with the introduction of the device? Has the device been integrated with existing activities or introduced as a part of entirely new programming or some combination of the two?

7 .3 .2 Learning Opportunities How has the device influenced learning opportunities in the site? Have learning opportunities that existed at the site been influenced by the device? Do students use devices in ways that formally or informally support their learning?

7.4 Typical UseHow do staff and students typically use the device in their instruction? Were there any differences between how staff and students described their practice and what researchers observed in the classrooms?

7.5 CommunicationWas the device used to support the practice and development of communication skills? If so, please comment on how common this kind of use was and on the range of communication styles and formats . Please also note whether communication practice ever involved outside audiences .

7.6 Innovative UseDescribe any innovative uses of the device for providing learning opportunities . Examples include involving students in projects, asking students to collaborate, having students assess their own work or the work of peers, etc .

7.7 Extension of Learning Beyond the SiteDescribe any efforts to use the device to extend learning outside the site. Are students connecting with community members or following up with further use at home or in school?

Qua

lita

tive

Rep

ort T

emp

late


8. Changes in Teaching and Learning (for school-based integration efforts)In this section, summarize responses from teacher, student, and school leader interviews to describe the changes that are taking place as a result of the integration of the device . In each category below, please include (1) strong or unique examples in each category, (2) what is common or similar across schools, and (3) any important variations or exceptions across schools .

8.1 Changes in Teacher Attitudes Do the data from teacher and school leader interviews indicate any shift in teachers’ attitudes (toward teaching, students, the use of device, etc.)? Please elaborate with specific examples.

8.2 Changes in Instructional PracticesDo teachers and school leaders report any shifts in instructional practices that arise from the use of the device? If so, what are they?

8.3 Changes in Curriculum and Assessment Do the data from teacher and school leader interviews indicate any shift in curriculum and assessment practices? How so?

8.4 Changes in Students’ Learning Behaviors Do the data from student and teacher interviews indicate any shift in students’ attitudes (toward school, learning, the use of device, etc.)? Do the data indicate any changes over time in how students learn? Please elaborate with specific examples.

8.5 Changes in Policies and PoliticsDo the data indicate a shift in policies or politics at the school, regional, or national levels? Please elaborate with specific examples.

8.6 Unexpected or Emergent Changes Do the data indicate that the technology solution was deployed in a manner that was different from what was intended? Did participants’ experiences vary considerably from what was anticipated? Did participants use the devices in unexpected ways and did the use of the device bring about unanticipated changes in their attitudes, learning behaviors, etc.?

9. Other Information or ExamplesInclude any other important information or examples that do not clearly fit under any of the previous headings. Describe why the information or examples are significant for education technology integration research

Qualitative R

eport Tem

plate

For more information on Intel education programs and technology, visit: www.intel.com/education

1 We recognize that purposive or convenience approaches, which result in a sample of sites that are of known interest or easy to recruit, might be more appropriate for several integration contexts. Other, more random sampling strategies may not yet be warranted, given the scope of the research and costs associated.

2 Though many integrations will take place within school contexts, some will involve non-school institutions such as out of school learning environments, orphanages, and other places where students participate in organized activities. We use the terms community center and community learning center to describe this broad range of possible non-school settings.

3Thisprotocolmaybeusedwithminimalmodificationsforinterviewswithadministratorsinnon-schoolsettings. Copyright © 2014 Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries. 0514/AE/PDF Printed in USA Please Recycle 325242-003US

Documents

Guide to Monitoring eLearning Programs · localization of the tools. Although some research activities will ... Guide to Monitoring eLearning Programs 5 Step 1: Establishing the Setting