Embed Size (px)
Citation preview
PMTF "Level 1" Outcome Measures Project Guide
This PMTF Task Force is asking Libraries to assist with a field trial to capture and report outcomes and related service delivery measures. The pmtf.org portal offers participating libraries with secure access to record information required for the trial. If your library has yet to receive access, then please contact the PMTF task force for details about participating.
http://www.pmtf.org/
Field Trial Summary There are 7 outcome surveys relating to the following program types:
Civic/Community Engagement Digital Inclusion Early Childhood Literacy Economic Development Education and Lifelong Learning Job Skills Summer Reading
For the trial, Libraries may selectively choose to capture outcomes for any of theprograms (listed above) at any or all outlets.
Programs are delivered by the library and basic information about each is captured usingthe Scheduler/Attendance form specific to each program type. The information capturedincludes the following:
a. Program Nameb. Locationc. Number of sessionsd. Date/Time (of final session if more than one)e. Length (time duration of program delivery)f. Attendance (distinct individuals attending)g. Staff notes (optional details that may help explain the results)
Programs may be a single session or comprised of a series of related sessions (e.g.,Reading Club).
Outcomes measurement is related to each program delivered (whether a series or singlesession). This information provides context for related outcomes.
Libraries can offer online survey capture or distribute paper copies to capture outcomesappropriate to each scheduled program.
Paper versions of the outcomes survey forms are available for each program type.Select the PDF version of the survey, specific to each program type. [note: Worddocuments are available upon request, should you wish to customize the paper surveys]On these survey forms you should record information specific to the program attended,
a. Program Nameb. Location
1 of 13
c. Date/Time As programs are scheduled, a unique URL is generated that can be shared with
participants. If your library prefers to capture outcome responses online directly fromprogram attendees, then just provide them with the unique URL. Each unique URLincludes all the details (date, time, name, etc.) about the program. Users are limited toone outcome survey response per session.
When distributing paper surveys, the program schedule and attendance details can all beentered at the same time, but must be entered before any survey responses can becaptured. Previously scheduled programs are available for review online using theAttendance form link.
The library is provided with a generic Survey link for each program type. These links aredesigned for library staff to enter outcomes responses on behalf of one or moreparticipants. You should not use this link until the related program has been scheduled.Previously scheduled sessions will appear as select options..
The PMTF Portal …
Authorized users should login to the PMTF portal using their provided username/password http://www.pmtf.org/
Once authenticated with their username/password, users can access the various programspecific forms, as listed on the secure landing page.
Program Schedule and Attendance App name and schedule (date time) details (duration, notes) and attendance
Attendance details (duration, notes) and attendance access to list of previously scheduled programs and details (sortable, searchable,
editable, downloadable) ability to add/update attendance and other data in the grid
Outcomes survey survey links, specific to each program, for internal staff use
Print Ready Survey download and print PDF version of survey
Unique Survey Links list of links and associated program details, that can be shared with attendees
Access to results (sortable, searchable, downloadable)
2 of 13
Program Types / Surveys The 7 Outcome surveys relate to programs that ...
Encompass common objectives/themes Provide similar outcomes for those that participate in these programs
Program Schedule A program must be scheduled before survey results can be captured/entered Program details are used as contextual information enabling views/reports of results …
relative to specific program session/series in combination/comparison with other occurrences of the same program type
(e.g., Early Childhood Literacy) in conjunction with attendance to determine survey response rates in conjunction with attendance to extrapolate outcomes in conjunction with duration to measure outcomes/hour type key ratios etc.
A program Scheduled must be entered prior to program delivery if capturing responsesonline from participants, or anytime before capturing survey responses (i.e., before orafter a program session/series has concluded). If entering all information afterconclusion of the program, then all scheduled details including attendance, duration andnotes can be entered at the same time.
Scheduling a program generates an unique URL, for optional use when capturing surveyresponses directly from participants.. This unique URL can only be used once per
3 of 13
session. If staff are capturing multiple results online, they should use the provided Survey link and not this unique URL.
Note: The unique survey URLs (provided to participants responding directly online) are only generated after a program is scheduled. The Unique URL links back to each scheduled program.
Using the Schedule App Select the “Schedule” link appropriate for the program type [note: the Program type and
Survey ID (for the outcomes survey) are preset] Select the delivery location (outlet where the program is offered) Select none if a virtually
delivered program that is not attributable to any specific location or when a program series spans multiple locations)
Provide the number of distinct program sessions relating to the program (see Program sessions vs. Program series for more information)
Provide the total duration of program delivery (e.g., 30 minutes = 30m; 1 hr 30 min = 1h 30m; or alternately 90m) Ultimately all results are converted into a minutes value, but for convenience, when a program spans multiple hours or days, you can just enter to total hours of program delivery. A program providing 8 hours on each of 2 days, do not enter 2d (2 days), instead enter 16h (16 hours). [Note: Decimal values are not supported. A program lasting 3.5 hours should be entered as 3h 30m.]
Provide date and time of program delivery (or program completion date/time if the series of sessions spans multiple days). If the same 2 hour program is offered simultaneously in two different physical locations, then count them as two distinct sessions each with 2h (2 hours) duration. If a program consists of two 2 hour sessions on the same day to the same audience, then you can count this as one scheduled program lasting 4h (4 hours).
Provide the event time (this helps to distinguish specific program session that occur on the same day e.g., multiple distinct Job Coaching oneonone sessions)
Optionally add the description of the program (as appropriate to provide context for the results)
4 of 13
Example: Schedule form for Early Childhood Literacy programs
This step in the process…
Identifies each unique occurrence of a program session/series for which you will capture outcomes (appropriate to each Outcomes survey)
Provide contextual information about the program session/series Creates a Unique ID identifying the the program that participants attended for the online
Outcomes survey. See Unique Survey URL by Program.
Program sessions vs. Program series Programs may be offered in a distinct single session (e.g., Job Coach) or as series of related sessions (e.g., Summer Reading).
Program Session Each occurrence is an single event: date, time, place and attendee(s) Concludes on the date and time specified Total hours of program delivery equivalent to the duration of the event
Program Series Comprised of multiple program sessions Concludes on the date and time specified Program sessions in the series may or may not have a distinct purpose
5 of 13
Program sessions in the series may occur more than once Total hours of program delivery is calculated as the sum from all program sessions in the
series Since attendees may attend one, some or all program sessions in the series, the total
attendance should be the total number of distinct attendees that participated (i.e., the distinct number of individuals that “could” respond to the survey). For example, if 20 people showed for the first of a two part session, and only 18 of 20 for the second session, then the total attendance is 20 …. not 20 + 18.
Attendance Includes a grid view of previously scheduled programs Select the appropriate entry to update the number of attendees (i.e., this is the number of
unique participants) Add program delivery notes, as applicable, pertinent to the delivery of the program
Note: The distinct attendance (number of participant(s)) represents the potential number of
respondents for the survey(s) The actual respondent count is realized after the survey(s) are completed
Adding attendance and notes Select the “Attendance” link relative to the program type
Before adding scheduled the program, you can add the program details and program
attendance and notes simultaneously through the Attendance form. Scroll down to the bottom of the form past the “Program delivery notes” question to find
the grid view for the previously scheduled programs, as shown.
6 of 13
Attendance grid view
Programs are listed in reverse chronological order in which they were added Rows can be sorted by column. Click on the column header to sort by that column. Search records. Select the magnifying glass icon, as shown.
Search scheduled programs
7 of 13
Add Attendance and Notes - Previously Scheduled Program Select the row that contains the program Select the pencil icon to edit the record to open the Edit Record dialog box Update the attendees count and any pertinent delivery notes Submit the update, as shown...
Edit Record
8 of 13
Schedule and Add Attendance Simultaneously If needed, a program can be scheduled and attendance added simultaneously through
the Attendance view. Select the Program Schedule and Attendance header to open the collapsible section, as
shown. Input the program details and attendance, including any pertinent notes.
9 of 13
Outcomes Surveys Specific Outcomes surveys are provided for each of the seven program types Surveys measure the anticipated outcomes that participants are likely to realize from
participating in a program Survey’s are ideally administered immediately upon program completion Scheduling a program generates a Unique ID to create a unique URL link for the
program, referencing… program type, program name, outlet (location delivered), distinct sessions, program completion/delivery date, and time of event
10 of 13
Unique Survey URL by Program Scheduled programs are each given a Unique Id (GUID) on submission of the “Schedule”
form. Lookup the Unique URL by program type in the Patron Survey list (see example below). Copy these links to distribute and/or promote the survey to the program participants.
Patron Surveys
Program details relative to the Unique ID (GUID) are prefilled automatically in the survey form
Participant does not need to recall the details of the program and provide this information as part of their survey response
URLs can be given to participants at the conclusion of the program or sent by email, if possible
11 of 13
About the Surveys
Program Outcomes surveys Conceived and constructed by the Performance Measures Task Force (PMTF) Questions and response options cannot be altered during the Level 1 trial Questions or comments about the survey(s) can be sent to
Administering the surveys The goal of the PMTF Level 1 trial is to measure the immediate outcomes realized by the participants in the programs delivered by the library. It is important, therefore, to engage the participants as soon as possible upon conclusion of the program. You can distribute paper forms or provide the unique survey URL to participants during the final session or have staff available to capture participant responses. The Library should keep track of the distinct number of participants for each program.
Methodology
Surveys can be conducted ... Online by providing participants with the unique outcomes survey URL for the program
they attend; Using provided paper copies of the survey (remember to add the program name,
date/time and location) distributed to the participants before/during or immediately following the final program session ; or
Interview each participant immediately following the program using the survey URL specifically designed for staff entry of responses (allows multiple consecutive survey submissions from the same user);
Online surveys - Unique survey URLs specific to each scheduled program can be created, see Unique
Survey URL by Program Participants can submit responses directly relative to the program they attend. No staff
input needed. Participants invited to take the survey…
URL provided to participants URL link available on a web page that participants are directed to If applicable/available, send an Email to participants including the unique URL link. Internet connected device (e.g., iPad) with survey URL preloaded lent to
participant or on behalf of the participant via staff intermediation.
Paper surveys - Generic paper versions of the surveys for each program type are available in the secure
portal. Select the “Paper” link for the survey that corresponds to the completed program type.
12 of 13
Remember to write the Program Name on the paper survey form and add the program delivery location (Outlet), Date and Time of program delivery, at the bottom of the survey page in the space provided. This same information is required in the Program Schedule and Attendance form and serves as a reference for data entry.
Distribute the paper survey documents to attendees before they leave. The surveys require only a few minutes depending on length of openended comments provided.
Collect the surveys and input the responses through the Survey links provided on the PMTF portal. Each program type has a unique survey input link. It is important to select the correct link for the program. Note that surveys cannot be entered if the program was not already added to the program schedule and attendance form. When you access the Survey link you can select the appropriate entry from the list of previously scheduled program s. Remember to add the attendance figures and any session notes using the Attendance link or directly in the Schedule form, if not previously setup.
Interviews with paper survey copy - Conduct oneonone with each participant Read questions and record responses (e.g., For the following statements which I will
read to you, you have 6 response choices: Strongly Agree; Agree; Neither Agree nor Disagree, Disagree, Strongly Disagree; or Not Applicable. For the statement [Read Statement 1], do you [Read Options].
Do not coach responses Do not rephrase the statements or response options. You may repeat the statements
and choices. If the respondent does not understand what is being asked of them, skip the question
entirely. Do not select not applicable in this case. Selecting not applicable implies the statement did not apply versus the question was not understood.
Survey input links for the program can be found in the Program Manager page.
Reports and Outcomes Survey Results Upon conclusion of the fieldtest, Counting Opinions will review and analyze the results of
the outcomes surveys and will report findings and recommendations to the Performance Measures Task Force. Libraries participating in the trial can access and download their own data at any time.
© 2014 Counting Opinions (SQUIRE) Ltd.
13 of 13
