Whitepaper:  UNESCO  Exam  Development  Microsoft  Certified  Educator  

 

The  Exam  Development  Process:  Microsoft  Certified  Educator    

Introduction  This  paper  provides  information  about  how  the  certification  exam  content  that  aligns  with  the  Technology  Literacy  approach  of  the  UNESCO  Competency  Framework  for  Teachers  (UNESCO  ICT-­‐CFT)  was  created.        

Microsoft’s  Assessment  and  Certification  Exam  (ACE)  team  worked  with  a  global  exam  development  vendor  to  create  this  exam.    The  Microsoft  development  team  consisted  of  a  Senior  Psychometrician,  a  Senior  Content  Development  Manager,  and  an  experienced  Project  Manager.    The  vendor  development  team  consisted  of  two  Lead  Content  Development  Managers,  a  Senior  Editor,  and  Subject  Matter  Experts  (SMEs)  with  deep  experience  in  the  integration  of  ICT  in  teaching  and  learning,  recruited  from  around  the  world.  

Exam  Item  Development  Methodology  &  Process    Microsoft  is  committed  to  creating  high  quality,  legally  defensible,  and  psychometrically  sound  exams.    To  support  this  goal,  our  methodology  follows  industry  standards  and  best  practices  and  incorporates  input  and  expertise  from  a  geographically  diverse  set  of  experts.    SMEs  were  identified  by  UNESCO  education  sector  and  ICT-­‐CFT  partners  while  others  were  recruited  from  academic  institutions  around  the  world.      

The  exam  item  development  process  proceeded  as  follows:  

• SMEs  attended  an  in-­‐person  item  writing  session  in  Montreal,  Canada.  This  session  was  facilitated  by  an  experienced  exam  content  development  vendor  and  lasted  for  two  weeks.  

• Every  morning,  each  SME  drafted  3-­‐4  exam  items  that  aligned  with  the  exam  specification.    Items  were  then  reviewed  by  the  Content  Development  Manager  to  ensure  that  they  mapped  appropriately  to  the  objective  and  adhered  to  Microsoft’s  item  writing  guidelines.    

• In  the  afternoon,  two  types  of  reviews  occurred:  Peer  and  Group.  

o During  Peer  Review,  each  SME  reviewed  another  SME’s  items  and  offered  suggestions,  comments  and  revisions  for  improvements.    These  changes  were  incorporated  before  the  item  was  sent  for  Group  Review.      

o During  Group  Review,  the  facilitator  reviewed  the  items  with  all  SMEs.    The  item’s  technical  accuracy,  cognitive  level,  real  world  value,  and  alignment  to  the  content  domain  were  discussed  as  appropriate,  and  the  item  was  modified  accordingly.    All  modifications  were  made  by  the  facilitator  after  consensus  was  reached  with  the  SMEs.      

• On  the  final  day  of  the  item  development  session,  each  SME  was  assigned  a  pool  of  items  to  which  they  added  the  item  rationale.    This  process  resulted  in  an  additional  review  of  the  item  based  on  comments  and  queries  by  the  SME  writing  the  rationale;  all  of  these  questions  and  comments  were  discussed  using  a  process  similar  to  the  Group  Review  described  above.      

• If  SMEs  could  not  reach  consensus  on  the  technical  accuracy,  appropriateness,  and  relevance  of  the  item,  the  item  was  rewritten  by  the  SMEs  using  a  collaborative  process.      

Beta  Test  Phase:  Testing  Items  for  Psychometric  Soundness    The  purpose  of  the  beta  test  phase  is  to  ensure  that  the  items  are  fair  and  psychometrically  sound  prior  to  scoring  candidates.    Psychometrically  sound  items  are  those  that  are  neither  too  easy  nor  too  difficult,  differentiate  between  high  and  low  performers,  and  are  reliable  measures  of  the  content  domain;  only  psychometrically  sound  items  will  be  included  on  the  live  version  of  the  exam.  Because  this  psychometric  information  can  only  be  obtained  by  having  candidates  sit  the  exam,  an  unscored  beta  exam  is  administered  to  gather  the  necessary  data.  This  data  ensures  the  quality  of  resulting  certification  exam.  

Process  The  beta  exam  pool  is  administered  to  an  invitation-­‐only  audience  that  is  similar  to  the  target  audience.    Ensuring  that  the  beta  audience  and  target  audience  are  similar  is  required  to  obtain  meaningful  psychometric  results.    

The  beta  exam  will  be  available  for  interested  candidates  until  the  live,  scored  version  is  published.    

Scoring    Candidates  who  take  the  beta  exam  will  not  receive  a  score  or  any  information  on  their  passing  status  until  the  psychometric  analysis  occurs.  The  process  of  obtaining  sufficient  data  and  psychometric  analyzing  this  data  typically  takes  approximately  6-­‐8  weeks;  however,  because  the  psychometric  analysis  cannot  begin  until  sufficient  data  has  been  collected,  this  estimate  is  dependent  on  the  amount  of  time  required  to  have  500  beta  candidates  take  this  exam.      

(Note  that  500  is  the  minimum  number  needed  for  meaningful  psychometric  analysis  for  this  certification  exam;  the  required  number  of  beta  participants  may  increase  or  decrease  for  other  certification  exams  depending  on  the  number  of  items  administered  during  each  delivery  of  the  beta  exam.)  

Once  sufficient  data  has  been  collected,  the  results  will  be  analyzed.  Items  that  are  not  psychometrically  sound  will  be  removed  from  the  item  pool.  At  this  time,  all  beta  tests  will  be  scored.  Certifications  will  be  awarded  to  beta  candidates  as  appropriate  based  on  those  scores.  

What  Candidates  Should  Expect  During  the  Exam  Each  delivery  of  the  exam,  including  the  beta  exam,  will  contain  approximately  50  items.  Note  that  this  is  subject  to  change  based  on  desired  exam  time  and  the  psychometric  performance  of  the  content  over  time.  The  item  pool  contains  both  case  study  items  and  multiple  choice  items.      

Candidates  will  be  given  150  minutes  to  complete  the  beta  exam  to  ensure  that  they  have  sufficient  time  to  complete  the  exam.  We  will  use  the  information  obtained  during  beta  to  set  the  final  exam  time.  Our  target  exam  time  for  the  live  exam  is  120  minutes,  but  the  amount  of  time  provided  will  be  based  on  the  time  that  it  takes  to  complete  the  number  of  items  (estimated  at  50)  needed  for  a  valid  and  reliable  exam.  

Sample  Questions  The  case  study  exam  format  uses  scenarios  that  simulate  how  teachers  might  use  ICT  resources  as  they  prepare  for  and  conduct  their  classes.  A  case  study  model  enables  us  to  test  a  teacher’s  ability  to  analyze  and  synthesize  information  to  make  decisions.    Case  Studies  contain  separate  sections  (e.g.,  Learning  and  Teaching  Environment;  Student  Objectives;  Professional  Development  Objectives);  candidates  answer  several  questions  based  on  information  provided  in  the  case  study.      

The  following  samples  are  representative  of  items  candidates  can  expect  to  see  associated  with  each  case  study.  Please  note  that  these  items  are  samples;  disclosing  actual  items  would  compromise  the  security  of  the  exam.        

Sample  1  Your  students  have  accomplished  the  learning  objectives  and  submitted  their  work.      You  need  to  show  the  work  to  parents  at  the  next  parent  teacher  conference.    You  want  to  display  the  work  in  a  continuous  loop  for  three  hours,  on  a  computer  screen.        Which  resource  should  you  use  to  display  the  student  work?  

□ Word  processing  software  □ Web  publishing  software  □ Presentation  software  □ Desktop  publishing  software  □ Picture  editing  software  

 

Sample  2  What  ICT  skills  do  your  students  need  to  develop  to  meet  the  learning  objectives?  

□ How  to  use  a  database  to  store  information.  □ How  to  carry  out  effective  Internet  searches.  □ How  to  download  files  to  a  removable  storage  device.  □ How  to  edit  photos  in  an  image-­‐editing  package.  

 

Global  Relevancy  Ensuring  global  relevancy  has  been  a  critical  goal  throughout  the  project  to  date.    The  UNESCO  ICT-­‐CFT  and  all  design  specifications  were  created  with  the  input  of  SMEs  from  around  the  world.        

The  SMEs  chosen  to  develop  the  exam  items  were  also  recruited  from  an  international  pool.    This  was  intentional  and  intended  to  ensure  global  representation  and  perspectives.    The  item  development  team  consisted  of  three  experts  from  Europe,  one  expert  from  North  America,  one  expert  from  Africa,  and  one  expert  from  Asia.        

Local  Contextualization  The  development  team  adhered  to  the  Microsoft  Globalization  standards  that  are  currently  applied  to  all  Microsoft  Certification  exams.    Compliance  with  these  standards  ensures  a  consistent  and  predictable  candidate  experience  across  all  countries  while  also  maintaining  the  intended  reliability  of  the  certification  credential.    Modifying  exams  at  a  local  level  is  not  currently  supported  as  doing  so  may  compromise  the  validity  of  the  certification.