WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005

WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005


WP4:Pilots with the Integrated System

Validation of the Project UPF 8, ALTRAN 7, LogicaCMG 3, OUNL 3, GIUNTI 5, CERTH 4, SU 4, SYN 2

Dai Griffiths, Josep Blat, UPF


Aims of the (integration) activity

• To provide realistic scenarios and corresponding pilots which constitute an effective environment for evaluation of the project, whereby the systems resulting from the research and development can be challenged and validated

• To provide the overall project evaluation strategy


Tasks, deliverables, timetable (I)

• T1. Define initial pilot scenarios (m6)• T2. Define the pilot evaluation plan (m6)• T5. Develop an overall evaluation plan (m6)• D4.1: Report (m 6) on:

– scenarios and pilots definition– user requirements and validation criteria for the

overall project– evaluation strategies and initial evaluation plan

implementation


Tasks, deliverables, timetable (II)

• T3. Set up the initial pilots (m6-12)• T4. Validate the initial system through the evaluation

of the Digital Cinema pilot (m12-18)• T6. Generate user requirements for cycle-2 pilots

(m12-18)• D4.2: Report on results of first evaluation of pilots

(month 18)– Suggested interim report on m12 to submit for the review


Pilots envisaged

• Pilot 1 Training for digital movie production (UPF)

• Pilot 2 Training for continuous education of health workers (UPF)

• Pilot 3 Unesco-IHE – Institute for Water Education (?)

• Pilot 4 The Antwerp Lifelong Learning City (?)– Request an initial description


Pilots (I)

• T1. Define initial pilot scenarios (up to m6)– Pilot 1: Digital movie production

• Partners (?)– Visual effects and other UPF postgraduate courses– Others from possible partners: Mediapro, Brainstorm, Thomson, Barco,

XDC, Filmlight, DVS, Pandora (MediaSalles, ...)– Commitments definition: elaborate on association document

• User requirements, validation criteria, evaluation• Target groups definition: learning providers, learning designers,

content, tutors, learners ... availability: STRESS THE IMPORTANCE OF A COMMUNITY OF USERS

• Plan for the implementation: define competencies based syllabi, ... INSERT HERE MATERIAL FROM IP-RACINE – DISCUSS TRANSFORMATION INTO COMPETENCE ORIENTED – DISCUSS THE CHALLENGES POSED BY THE PILOT

– Establish a model for the other pilots


Pilots (II)

• T1. Define initial pilot scenarios (up to m6)– Pilot 2: Training for continuous education of

health workers• Partners (?) Hospital Clínic, Doyma (others)• Gastroenterology/hepatology, Pediatrics, Colo-rectal cancer

screening, ...• User requirements, validation criteria, evaluation• Target groups definition: learning providers, learning

designers, content, tutors, learners ... availability


Pilots (III)

• T1. Define initial pilot scenarios (up to m6)– Pilot 3: Unesco-IHE – Institute for Water

Education• ?


Pilots (IV)

• T1. Define initial pilot scenarios (up to m6)– Pilot 4 The Antwerp Lifelong Learning City

• ?


Evaluation planning and initial implementation

• T2. Define the pilot evaluation plan, specifying the validation criteria and the strategies for evaluation (up to m6)

• T5. Develop an overall evaluation plan (up to m6)• Validation focus along cycles

– 'proof of concept' during the first cycle through the Digital Cinema pilot

– 'usability' during the second cycle through Health Workers pilot, the UNESCO-IHE pilot, the Antwerp Lifelong Learning City pilot, and an extension of the Digital Cinema pilot.

– ' wider applicability and sustainability' during the final cycle.


Evaluation planning (I)

• T5. Develop an overall evaluation plan (up to m6)– objectives addressed (including gender issues),– criteria and indicators:

• quality criteria• effectiveness in context• technical criteria

• components and functions to be assessed• methodologies in context of the project• the different actors and “reference users” in the process and

their tasks• the coordination and procedures, defining a timeline for

evaluation


Sample evaluation grid from TA

Activities Aspect RTD activities

Technical Design & Implementation activities

Pilots & Validation activities

Pilots & Validation activities

experiments Experiments pilots demonstrators

Objective Technical verification and validation of the functional and usability aspects of the tools

Verification of the integration and the interoperability between the components

Validation of the environment integrated and of the individual components. Focus:- functionality- operationally- usability

Validation of the environment integrated and of the individual components. Focus:- sustainability

Feeds into The concerning activities

Technical Design & Implementation + Aspect RTD activities

Technical Design & Implementation + Aspect RTD activities

Requirements & Analysis activities

Period ‘continuously’ after completion of each integration loop: 3 times

after the conclusion of the experiments of the first two integration loops: 2 times.Round 1: small scaleRound 2: large scale

after the conclusion of the experiment of the final integration loop: 1 time


On users/actors and methodologies

• Users / actors– learning providers, learning designers, tutors, learners ... (different

from different areas?)

• Methodologies to be used:– Ethnographic study, Interviews, Focus groups, Questionnaires,

Observations, Experiment, Peer review, Think-alouds, User diaries, System log data, Textual data, Cost analysis, Reviews, Usability inspection, ...

• Components / areas (to be defined for a good plan)


Evaluation planning in detail

• T5. Develop an overall evaluation plan (continued)– more detail needed for planning and initial implementation– evaluation contact in each organisation – not only this WP

partners– define components to be evaluated; provide initial evaluation

sheet for each component (model provided)– Initial scenarios for pilots probably needed to plan for

the whole period– UPF in charge of coordination and assistance, every partner

responsible of carrying out their own evaluation, evaluation committee to supervise overall evaluation and validation

– Everybody to send a list of components, and an evaluation sheet for each component


Evaluation sheet modelComponent to be evaluated Areas to be assessed

Parameters to be assessed

Actors / partners

Role of the actors

Evaluation procedure / material

Starting date

Immediate action (and date)

Intermediate action (and date)

End date

Conclusion (general / for each area)

Overall comments

Evaluation committee comments


Pilot evaluation plan and implementation

• T2. Define the pilot evaluation plan, specifying the validation criteria and the strategies for evaluation (up to m6)– Define an implementation plan for the first pilot – Model for the coming plans (which have to be

initially defined)


Deliverable reminder; contributions

• D4.1: Report (m 6) on:– scenarios and pilots definition– user requirements (?) and validation criteria for the overall project– evaluation strategies and initial evaluation plan implementation

• Contributions of the different partners (to be defined now for the first 6 months of the project)– OUNL ?– ALTRAN ?– LogicaCMG ?– GIUNTI ?– CERTH ?– SU ?– SYN ?– Anybody missing? Any contribution missing?


Evaluation representatives (all partners)

• OUNL ?• ALTRAN ?• LogicaCMG ?• GIUNTI ?• CERTH ?• SU ?• SYN ?• L3S ?• INSEAD ?• UB ?• UvA ?• SURF ?• Evaluation committee?


Reminder of coming tasks

• T3. Set up the initial pilots (m6-12)– Initial pilot on Digital Cinema; start earlier if possible to be able to

validate; otherwise, make it a phased pilot, where validation can start early

• T4. Validate the initial system through the evaluation of the Digital Cinema pilot (m12-18)– Phased work or earlier start

• T6. Generate user requirements for cycle-2 pilots (m12-18)– On the basis of early work on pilots definition

• D4.2: Report on results of first evaluation of pilots (month 18)– Suggested interim report on m12 to submit for the review


Ourselves

• Dai Griffiths (WP manager) [email protected]

• Josep Blat (in charge of DC pilot) [email protected]

• Sergio Sayago (evaluation specialist) [email protected]

• and others!


By way of summary

• Evaluation, validation, user requirement activities are probably everybody’s task

• Components and overall evaluation/validation strategies to be defined: contributions expected from everybody; not an easy task

• Pilot 1 (Digital Cinema) to be planned and carried out during first period

• Pilots 2, 3, 4 to be planned during m12-18; but initial scenarios important for the whole user reqs, validation, evaluation planning due by month 6. A second updated deliverable by month 12?

• And don’t forget the periphery of the project! Possible extra pilots, associates feedback ...

Documents

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005