1  

     

 2016  

NSF  Pan  REU  PI  Workshop  

Holiday  Inn  Rosslyn  at  Key  Bridge  

Arlington,  Virginia  

April  28  –  30,  2016    

   

Workshop  Report    

This  report  was  partially  funded  by  the  National  Science  Foundation  (Grant  Number-­‐  to  PI  names).    Any  opinions,  findings,  and  conclusion  or  recommendations  expressed  in  this  material  are  those  of  the  

authors  and  do  not  necessarily  reflect  the  view  of  the  NSF  or  the  US  government.                    

     

2  

Table  of  Contents  EXECUTIVE  SUMMARY  .....................................................................................................................................................  3  RECOMMENDATIONS  /  ACTION  PLANS  .................................................................................................................................................  3  

THE  PAN  REU  PLANNING  COMMITTEE  .......................................................................................................................  4  PAN  REU  PI  WORKSHOP  HISTORY  ...............................................................................................................................  4  2016  PAN  REU  PI  WORKSHOP  OVERVIEW  ................................................................................................................  5  WORKSHOP  GOALS  ...................................................................................................................................................................................  5  ACTIVITIES  ..................................................................................................................................................................................................  5  

2016  WORKSHOP  NOTES  AND  SUMMARIES  .............................................................................................................  6  POSTER  SESSION  .......................................................................................................................................................................................  6  NETWORKING  DINNER  .............................................................................................................................................................................  6  NSF  REU  PROGRAM  OFFICER  PANEL  ...................................................................................................................................................  6  BEST  PRACTICES  BREAKOUT  DISCUSSIONS  ..........................................................................................................................................  8  Cohort  Building  ......................................................................................................................................................................................  8  Mentor  Training  ....................................................................................................................................................................................  9  Program  Evaluation  .........................................................................................................................................................................  10  Participant  Tracking  ........................................................................................................................................................................  11  REU  Leadership  Group  .....................................................................................................................................................................  12  Career  Development  for  Students  ...............................................................................................................................................  13  Recruiting  and  Selecting  to  Broaden  Participation  ............................................................................................................  14  Common  Application  Deadline  .....................................................................................................................................................  15  

SHORT  TALKS  ..........................................................................................................................................................................................  16  GEO:  Val  Sloan  and  David  Fields  .................................................................................................................................................  16  CHE:  Linette  Watkins  ........................................................................................................................................................................  16  BIO:  Alan  Berkowitz  ..........................................................................................................................................................................  16  AST:  Kathy  Eastwood  .......................................................................................................................................................................  17  

COMMON  REU  ASSESSMENT  &  EVALUATION  TOOL  .......................................................................................................................  18  Common  Assessment  &  Evaluation  Tool  Disciplinary  Group  Discussions  .................................................................  18  

ACTION  PLANNING  BREAKOUT  SESSION  SUMMARIES  ....................................................................................................................  19  Developing  Collaborations  with  MSI’s  ......................................................................................................................................  20  Developing  Collaborations  with  Community  Colleges  .......................................................................................................  20  Formalizing  the  Pan  REU  Community  and  Development  of  a  Pan  REU  Website  ...................................................  20  Informing  the  Development  of  Long-­‐Term  Tracking  Tools  .............................................................................................  21  Communicating  with  REU  Stakeholders  About  Evaluation  Criteria  and  Metrics  for  Success  ..........................  22  Strategies  for  Broadening  Participation  .................................................................................................................................  23  How  Can  the  REU  Group  Inform  the  Agenda  and  Priorities  for  Congress  .................................................................  24  

APPENDIX  ...........................................................................................................................................................................  26  APPENDIX  1:  WORKSHOP  AGENDA  ....................................................................................................................................................  27  APPENDIX  2:  WORKSHOP  ATTENDEES  ..............................................................................................................................................  28  APPENDIX  3:  WORKSHOP  POSTER  PRESENTATIONS  ......................................................................................................................  31  APPENDIX  4:  ACTION  PLANNING  BREAKOUT  SESSION  NOTES  .....................................................................................................  33  APPENDIX  5:  WORKSHOP  EVALUATION  RESULTS  ...........................................................................................................................  41  2016  Pan-­‐NSF  REU  Workshop  Pre-­‐survey  ..............................................................................................................................  41  2061  Pan-­‐NSF  REU  Workshop  Post-­‐survey  ............................................................................................................................  52  

     

3  

Executive  Summary    A  workshop  for  principal  investigators  of  Research  Experiences  for  Undergraduates  (REU)  site  programs  funded  by  the  National  Science  Foundation  was  held  on  April  28-­‐30,  2016  at  the  Holiday  Inn  Rosslyn  in  Arlington,  Virginia.  The  workshop  theme  was  “Leveraging  Excellence  Through  Collaboration  Across  REU  Programs.”  It  brought  principal  investigators  from  across  the  NSF  Divisions  together  to  network,  share  best  practices  and  identify  and  develop  recommendations  for  potential  Pan  REU  projects.  In  this  report  we  document  key  recommendations  and  action  plans  that  emerged  from  the  meeting,  as  well  as  document  the  activities,  discussions,  and  presentations  that  occurred.  Our  primary  recommendation  is  to  strengthen  the  Pan  REU  community  through  the  creation  of  a  Pan  REU  leadership  team.  This  team  can  help  organize  future  PAN  meetings,  which  would  ideally  happen  more  frequently,  as  well  as  develop  resources  that  can  be  shared  across  disciplines,  and  create  collaborative  partnerships  with  other  organizations  supporting  undergraduate  research,  such  as  the  Council  for  Undergraduate  Research.  Additional  key  recommendations  are  to  include  the  PAN  REU  community  in  development  of  tools  for  participant  tracking  to  satisfy  the  America  Competes  Act,  and  to  take  greater  advantage  of  existing  reporting  data  for  documenting  REU  outcomes.    

Recommendations  /  Action  Plans  1.  Strengthen  the  Pan  REU  Community  

a. Officially  create  and  support  a  Pan  REU  Leadership  Council  b. Increase  frequency  of  Pan  REU  workshop  meetings  c. Maintain  a  Pan  REU  website  with  shared  resources  (e.g.,  assessment  tools,  PI  professional  

development  opportunities,  alumni  stories,  etc.)  d. Establish  sub-­‐committees  for  specific  interest  groups  (e.g.,  minority  serving  institutions,  

community  colleges)  with  ex-­‐officio  membership  on  the  Pan  Leadership  Council    e. Establish  partnerships  with  other  undergraduate  research  organizations  (e.g.  CUR)  

 2. Evaluation  of  REU  Programs  and  Tracking  of  REU  Participants  

a. The  Pan  REU  community  should  be  given  the  opportunity  to  educate  and  inform  the  process  by  which  NSF  fulfills  the  participant  tracking  requirements  for  the  America  Competes  Act.  

b. The  REU  community  needs  support  to  determine  how  to  define  and  evaluate  REU  outcomes  and  impact.    

c. The  REU  community  should  contribute  to  defining  REU  community  goals  and  performance  metrics    

d. The  existing  data  submitted  to  NSF  in  annual  reports  should  be  mined,  utilized  and  made  available  in  aggregate  for  evaluation  purposes  

e. An  NSF  ID  number  for  REU  applicants  or  participants  should  be  created  for  tracking  purposes    

3. Access  to  and  Quality  of  REU  Opportunities    a. Promote  the  demand  for  and  access  to  REU  opportunities  across  a  broad  population  of  students  b. Expand  the  student  conference  travel  award  program.  c. Offer  professional  development  opportunities  to  REU  program  directors  and  research  mentors  

to  promote  high  quality  programming  and  effective  mentor-­‐student  relationships.    

4. REU  Program  Solicitation  Revisions  a. Solicit  input  from  the  REU  community  when  revising  the  REU  program  solicitation  b. Continue  support  of  a  broad  diversity  of  specific  program  goals  and  approaches;  avoid  narrow  

or  rigid  criteria  for  success,  unless  there  are  clearly  documented  evidence-­‐based  practices.  

4  

c. Articulate  the  definition  of  broadening  participation    

5. Advance  Congressional  Awareness  and  Support  of  the  REU  Program    a. Individual  REU  sites  should  inform  members  of  congress  and  their  staff  about  the  REU  program  b. Develop  and  disseminate  best  practices  for  sharing  with  Congressional  members  and  staffers.  

i. Students  writing  letters  ii. Forming  relationships  with  Congressional  members  and  staffers  iii. Using  Twitter  (#NSFREU)  

c. Develop  a  1-­‐page  student  highlights  template  that  program  directors  can  use  when  communicating  with  congress.  

The  Pan  REU  Planning  Committee  The  planning  committee  was  formed  at  the  end  of  2015  and  then  held  bi-­‐weekly  teleconferences  for  three  months  preceding  the  workshop  in  April.  Members  of  the  committee  represented  the  different  divisions  that  fund  REU  programs  and  a  diversity  of  types  of  REU  programs.  Committee  members  worked  in  small  groups  to  take  responsibility  for  leading  different  portions  of  the  workshop.    

Name   Email   Institution   Discipline  Ian  Billick   director@rmbl.org   Rocky  Mountain  Biological  Laboratory   BIO  

Janet  Branchaw,  Co-­‐Chair   branchaw@wisc.edu   University  of  Wisconsin,  Madison   BIO  Karen  Buchmueller,  Co-­‐

Chair   karen.buchmueller@furman.edu   Furman  University   MPS/CHE  Debbie  Chachra   Debbie.Chachra@olin.edu   Olin  College  of  Engineering   EHR  Abhaya  Datye   datye@unm.edu   University  of  New  Mexico   MPS/DMR  Holly  Gaede   hgaede@mail.chem.tamu.edu   Texas  A&M  University   MPS/CHE  

Allison  Huff  MacPherson   allison@optics.arizona.edu   University  of  Arizona   ENG/EEC  Rachel  Kallen   rachel.kallen@uc.edu   University  of  Cincinnati   SBE  Charles  Liu   cliu@mail.csi.cuny.edu   CUNY  College  of  Staten  Island   MPS/AST  

Darren  Narayan   dansma@rit.edu   Rochester  Institute  of  Technology   MPS/DMS  Valerie  Sloan   vsloan@ucar.edu   UCAR   GEO/OCE  Steve  Turley   turley@byu.edu   Brigham  Young  University   MPS/PHY  Dan  Wubah   dwubah@wlu.edu   Washington  and  Lee  University   BIO  

Pan  REU  PI  Workshop  History  The  2016  workshop  was  the  second  workshop  held  for  NSF  REU  PIs  across  divisions.  The  first  was  held  over  10  years  ago  in  2005  and  focused  on  sharing  best  practices  across  REU  divisions.  

5  

2016  Pan  REU  PI  Workshop  Overview  

Workshop  Goals    

(1) Build  community  across  the  NSF  REU  disciplinary  PIs;  (2) Identify  areas  of  collaboration  that  could  increase  access  and  diversity  across  NSF  REU  programs;  and  (3) Begin  planning  for  collaborative  initiatives  by  identifying  leaders  and  forming  working  teams.  

 Workshop  activities  were  designed  to  provide  PIs  the  opportunity  to:  

1. network  and  share  best  practices  • Activities:  poster  session,  welcome  dinner  and  lunch,  “How  To”  panels  and  workshops  (e.g.  

recruitment,  logistics,  mentoring,  enrichment  activities,  assessment,  tracking,  etc.)    

2. gather  input  from  NSF  REU  Program  Officers  about  the  potential  for  pan  REU  projects  and  collaborations.    • Activities:  NSF  Program  Officer  panel    

 3. identify  best  practices  and  resources  that  could  be  coordinated  and  made  available  across  REU  

programs.  • Activities:  sharing  best  practices  breakout  sessions,  short  talks    

4. develop  action  plans  • Activities:  action  planning  breakout  groups  

 

Activities    

Poster  Session  -­‐  Workshop  participants  presented  posters  (Appendix  3)  highlighting  the  best  features  of  their  REU  programs  to  share  best  practices  and  models.      Networking  Dinner  -­‐  Workshop  participants  met  one  another  and  shared  experiences  and  best  practices  around  common  interests.    NSF  Program  Officer  Panel  –  Panelists  shared  their  vision  for  the  future  of  REU  programs  at  NSF  and  answered  questions  from  workshop  attendees.    Best  Practices  Breakout  Discussions  –  Workshop  participants  identified  topics  of  interest  before  the  workshop  and  were  assigned  to  breakout  discussion  groups  based  on  those  preferences.  

 Common  Assessment  and  Evaluation  Tool  Presentation  &  Disciplinary  Discussions  -­‐  An  update  on  the  development  of  a  new  platform  for  the  Biology  REU  Common  Assessment  Tool  was  presented.  Disciplinary  breakout  groups  discussed  the  pros  and  cons  of  using  a  common  assessment  tool  and  provided  feedback  on  the  new  platform  to  the  developers.    

6  

Action  Planning  Working  Groups  –  Small  groups  of  participants  met  on  the  last  morning  of  the  workshop  to  develop  action  plans  to  address  topics  of  interest  that  emerged  from  the  community  during  the  course  of  the  workshop  activities.    Workshop  participants  volunteered  to  take  notes  at  each  breakout  session.  In  addition  to  capturing  the  main  points  of  the  discussion,  they  recorded  feedback  and  recommendations  for  this  report.  

2016  Workshop  Notes  and  Summaries    

Poster  Session  The  poster  session  sparked  conversations  between  workshops  participants,  which  included  sharing  of  best  practices  such  as  effective  means  for  engaging  student  in  research,  assessing  program  successes,  and  tracking  of  participants.  A  call  for  posters  was  placed  on  the  workshop  web  page.  Twenty-­‐six  posters  were  received  and  displayed  on  Thursday,  April  28,  2016  from  4:00-­‐6:00  p.m.  in  the  atrium  of  the  National  Science  Foundation  building.  This  provided  an  opportunity  for  personnel  from  NSF  to  also  view  the  posters.    Posters  were  then  displayed  for  the  remainder  of  the  meeting  in  the  main  meeting  room  at  the  hotel.    The  focus  of  the  poster  session  was  to  highlight  unique  elements  of  broad  interest  to  PIs  of  other  REU  sites.  Presenters  were  asked  to  highlight  what  their  program  does  well  including,  but  not  limited  to:  

1. Unique  educational  aspects,  such  as  training  of  students  to  become  future  faculty,  professional  ethics,  research  skills  etc.,  and  how  your  program  succeeds  in  meeting  this  goal.  

2. Approaches  to  broadening  participation  -­‐  recruitment  of  underrepresented  groups,  recruitment  from  institutions  with  limited  research  opportunities,  including  two-­‐year  colleges.  

3. Assessment/program  evaluation  and  long-­‐term  tracking  strategies.  4. Outcomes,  publications,  patents  

Other  topics,  such  as  general  information,  research  focus,  program  structure,  were  welcomed.  

Networking  Dinner  The  dinner  was  designed  to  continue  the  networking  and  discussions  begun  during  the  poster  session.  Participants  sat  at  the  table  of  their  choice,  approximately  six  individuals  sat  together  at  one  table.  Prompts  were  provided  to  stimulate  discussion.  

NSF  REU  Program  Officer  Panel  Program  officers  were  invited  to  present  an  overview  of  the  REU  program,  its  future,  and  what  the  PI  community  can  do  to  assist  the  program.  The  program  officers  that  presented  were:  Fahmida  Choudhury,  SBE  (she  could  not  be  there,  but  sent  comments);  Kathy  McCloud,  PHY;  Ty  Mitchell,  CHE;  Sally  O’Connor,  BIO;  Mary  Poats,  ENG;  and  Lisa  Rom,  GEO.  The  questions  and  summarized  answers  are  given  below.    1. What  are  the  priority  areas  for  REU  in  your  area  and  what  gaps  do  you  see  in  your  REU  portfolio?  Biology:  NRC  reports  guide  priorities,  and  these  topics  are  sought  in  the  proposals.  They  have  been  able  to  fund  sites  relatively  quickly  after  the  introduction  of  cutting  edge  research  tools  (e.g.  CRISPR).  They  have  used  the  EAGER  mechanism  to  fund  cutting  edge  research,  and  alternative  ideas.  Biology  seeks  more  international  proposals  and  sites  in  emerging  areas  of  research,  including  INFEWS  (Intersection  of  Food,  Energy,  and  Water);  NEON  (The  National  Ecological  Observatory  Network);  BioMAPS,  at  the  intersection  between  disciplines.    

7  

Chemistry:  A  new  emphasis  has  been  to  link  LSAMP  and  REU  to  link  URM  students  seeking  research  opportunities,  with  programs  offering  those  opportunities.  Engineering:  Only  35  states  are  represented  so  improving  representation  is  a  goal.  Also,  there  is  interest  in  combined  REU-­‐RET  sites  and  sites  serving  veterans.    Geosciences:  The  primary  priority  is  to  increase  participation  by  students  from  underrepresented  groups.  GEO  is  also  interested  improving  retention  in  STEM  of  students  in  their  first  or  second  year  of  college  or  in  community  college.  In  GEO/OCE  (Ocean  Sciences),  PIs  are  encouraged  to  seek  the  participation  of  students  with  backgrounds  in  physics,  chemistry,  engineering,  and  geology.  The  EAR  Division  is  eager  to  see  REUs  implement  ethics  training,  and  to  see  proposers  consider  alternative  models  such  as  distributed  REUs  and  international  REUs.  The  AGS  Division  is  also  interested  in  international  sites  and  at  HBCUs.    All  of  the  Divisions  are  interested  in  seeing  REU  PIs  share  evidence-­‐based  practices.  Physics:  The  portfolio  includes  really  good  research,  including  at  large  facilities.  A  diversity  of  sites  is  funded,  including  sites  that  emphasize  early  in  career/later  in  career,  international,  and  distributed.  The  plan  is  to  continue  with  a  diversity  of  programs.  Institutions  should  go  with  their  strengths.  The  biggest  gap  is  a  lack  of  diversity  of  the  participants.  Social,  Behavioral  and  Economic  Sciences:  SBE  welcomes  proposals  from  all  areas  of  SBE  sciences,  plus  interdisciplinary  and  multidisciplinary  projects  as  well.  The  current  SBE  REU  Sites  portfolio  reflects  the  landscape  of  undergraduate  research  culture  in  the  country,  for  example,  undergraduate  research  is  common  in  archaeology,  psychology  etc.,  but  less  common  in  economics,  and  that  is  reflected  in  the  proposals  received.    2. What  thoughts  or  ideas  do  you  have  for  possible  Pan  REU  collaborative  projects?    Discussions  between  REU  Site  PIs  about  potential  areas  of  collaboration  between  different  fields  could  encourage  and  help  shape  some  pan-­‐NSF  REU  sites.  Some  of  this  already  happens.  For  example,  Cathy  McCloud  and  Fahmida  Choudhury  have  co-­‐funded  REU  Sites  at  the  Santa  Fe  Institute,  and  Lisa  Rom  and  Fahmida  Choudhury,  have  co-­‐funded  a  Smithsonian  REU  Site.      Areas  for  potential  collaboration  include  methods  to  track  student  applications,  development  of  an  ethics  curriculum,  sharing  of  lessons  learned  from  distributed  REU  sites,  and  development  of  consistency  and  uniformity  across  sites.  Additional  areas  for  collaboration  include  development  of  a  common  assessment  tool,  creating  a  travel  scholarship  fund,  developing  leadership  groups,  developing  ethics  training,  and  sharing  booths  at  scientific  conferences.      3. What  sorts  of  interdisciplinary  proposals  do  you  see,  and  how  is  the  review  of  proposals  that  have  

significant  interdisciplinary  content  handled?    

Fig.  1.  A  panel  of  NSF  REU  Program  Officers  answered  questions  from  the  REU  PIs  from  various  disciplines.    From  left  to  right,  panelists  were  Sally  O’Connor  (BIO),  Ty  Mitchell  (CHE),  Mary  Poats  (ENG),  Lisa  Rom  (GEO),  Kathy  McCloud  (PHY).    

8  

Currently,  the  mechanism  for  review  of  interdisciplinary  proposals  is  informal.  The  program  officers  can  share  proposals  after  review,  submit  them  for  review  by  two  panels,  or  transfer  them  to  another  group  for  review  with  the  promise  to  co-­‐fund  if  the  review  is  favorable.  PIs  can  choose  a  secondary  directorate,  which  will  alert  program  officers  to  possible  co-­‐funders.  The  program  officers  are  thinking  about  developing  a  more  formalized  co-­‐funding  mechanism.    The  entire  SBE  portfolio  is  interdisciplinary  in  the  sense  that  there  is  only  one  REU  Site  competition  for  the  whole  SBE  directorate.  Many  proposals  are  interdisciplinary  within  SBE,  or  sometimes  they  include  non-­‐SBE  fields  such  as  Geological  or  Biological  Sciences,  Engineering,  Computer  Science  etc.  Such  proposals  are  co-­‐reviewed  with  the  appropriate  panel,  or  receive  an  ad  hoc  (extra)  review  from  a  carefully  chosen  reviewer.    4. What  can  REU  PIs  do  to  support  the  REU  program  at  NSF?    REU  PIs  can  help  support  the  program  in  many  ways.  Name  your  REU  project  exactly  that:  an  NSF  REU  Site.  REU  Site  PIs  can  mentor  other  faculty  members  and  non-­‐academic  scientists  to  get  undergraduate  students  involved  in  their  work  and  mentor  new  REU  PIs.  Not  all  REU  programs  need  to  be  funded  by  NSF,  but  the  culture  of  undergraduate  research  can  be  spread  much  wider.  REU  PIs  can  develop  handbooks  of  best  practices.  Also,  REU  Site  PIs  can  help  us  by  volunteering  to  be  reviewers  for  the  program  and  encouraging  colleagues  to  do  so.  PIs  can  submit  stories  to  local  media  and  alert  the  NSF  press  office,  and  they  might  be  able  to  increase  the  impact  of  the  stories.  PIs  should  email  program  officers  lists  of  students  presenting  at  national  meetings.  PIs  should  serve  on  panels  and  encourage  colleagues  to.  PIs  should  submit  highlights  and  give  program  officers  data,  including  data  on  number  of  applicants  and  diversity  and  inclusion  information.  Encourage  students  to  send  letters  to  congressional  representatives.  PIs  can  provide  feedback  to  the  Program  Staff  at  NSF  on  any  aspect  of  the  program  as  they  see  fit.  Consider  studying  the  impact  of  the  REU  on  mentors  and  departments,  and  their  connections  with  colleagues  as  well  as  their  research.    Mentors  and  PIs  are  encouraged  to  disseminate  best  practices  with  faculty  in  order  to  improve  mentoring,  student  engagement,  and  valuing  the  impact  of  student  career  development  and  research  in  the  university  curriculum  and  culture.      

Best  Practices  Breakout  Discussions  Participants  were  surveyed  before  the  workshop  to  gauge  their  interest  in  breakout  discussion  topics.  Based  on  survey  responses,  participants  were  assigned  to  two  different  breakout  sessions  discussing  a  total  of  eight  topics,  with  most  topics  discussed  by  two  groups.  Prompts  were  developed  for  each  topic,  and  facilitators  from  the  steering  committee  were  assigned  to  lead  each  session.  Each  facilitator  was  asked  to  choose  a  scribe  to  record  the  main  discussion  points  to  summarize  the  discussion  for  the  group.  In  the  case  of  duplicate  groups,  reporters  met  before  report  back  to  consolidate  reports.    

Cohort  Building  Cohort  building  among  student  participants  and  faculty  mentors  is  an  essential  part  of  any  REU  program.  Many  directors  have  observed  that  positive  relationships  among  participants  and  mentors  enhances  research  productivity.  Various  programs  have  instituted  an  array  of  activities,  some  of  which  were  research  oriented,  others  social.  How  do  you  create  a  safe  and  inclusive  environment  for  your  participants?  

● Some  programs  have  student  participants  and  faculty  mentors  introduce  themselves  through  social  networks  such  as  Facebook  before  arriving  at  the  REU  site.  Once  participants  arrive  at  the  REU  site  many  programs  have  an  opening  breakfast,  lunch,  or  barbecue  where  students  and  mentors  can  informally  meet.  

What  cohort  building  activities  have  you  instituted?  

9  

● Having  participants  housed  in  the  same  dorm  is  a  great  way  to  build  community  interactions.  This  often  leads  to  shared  preparation  and  eating  of  meals.  Several  different  “organically  developing  activities”  were  used  at  various  sites  including  introductory  boot  camps,  field  trips,  and  camping  trips.  Research  enrichment  activities  included  seminars  in  ethics.  

What  social  activities  do  you  organize?  ● Daily  lunches  with  students  and  mentors  were  an  effective  means  of  community  building.  Others  ideas  

included  scavenger  hunts,  weekly  dinners,  seminar  presentations,  and  community  outreach  activities.    How  do  you  pay  for  them?  

● Events  related  to  the  research  with  required  students  participation  can  usually  be  funded  through  the  grant.  Social  activities  can  be  funded  by  the  REU  host  organization,  possibly  using  overhead  fund  from  the  REU  grant.    

What  challenges  do  you  face  in  community  building?  ● An  issue  that  arose  was  liability  and  the  safety  of  students.  In  many  cases  the  host  institution  may  have  

requirements  regarding  transportation  and  paperwork  that  needs  to  be  completed.  What  can  the  PAN  REU  community  do  to  help  in  this  task?  

● Continue  to  share  ideas  about  ideas  and  effective  practices  for  building  an  aspect  of  community  among  students  and  participants.    

Mentor  Training  Describe  programs  that  you  have  in  place  to  prepare  and  support  faculty  and/or  graduate  student  mentors.  

● Establish  individual  meetings  with  mentors  ○ BEFORE  the  program  starts,  establish  and  align  expectations  to  foster  mentor  ownership  of  the  

program;  ○ DURING  the  program,  provide  opportunities  for  feedback  and  address  mentoring  issues  as  they  

arise.  ● Develop  mentor  orientation  meetings  to  outline  the  expectations  of  mentors.  Share  these  expectations  

with  students  as  well.  In  certain  cases,  a  mentor-­‐mentee  contract  is  required.  ● Mentor  communities:  

○ Recruit  former  REU  participants  as  mentors  when  they  become  graduate  students  and  postdocs.  Undergraduate  peer  mentors  may  also  be  useful  at  small  liberal  arts  colleges.  

○ Create  discussion  groups  in  which  mentors  share  their  best  practices  (expertise)  with  one  another.  Use  case  studies  to  get  the  discussion  started.  

○ Use  formal  research  mentor  training  tools  such  as  the  Entering  Mentoring  seminar  to  provide  real  time  training  for  mentors  -­‐  note  that  small  groups  of  8-­‐10  mentors  works  best.  

○ Provide  credit  or  a  certificate  for  participating  in  the  community  or  training.  This  can  be  particularly  beneficial  for  mentors  who  are  graduate  students  or  postdoctoral  scholars.  

● Program  evaluator  presentations  highlighting  positive  comments  from  students  about  their  mentors.  ● Provide  and  train  mentors  on  how  to  use  rubrics  on  higher  order  thinking,  creative  thinking  and  

reflection  to  help  them  give  students  specific  feedback  (https://think.dasa.ncsu.edu).      What  challenges  do  you  face  in  preparing  and  supporting  mentors?  

● Mentors  are  not  always  willing  to  participate  in  mentoring  communities  or  formal  training,  often  because  they  believe  they  already  know  how  to  mentor  or  do  not  have  time.  PIs  need  to  be  mindful  of  returning  mentors  who  may  have  participated  in  previous  training  sessions.  

● It  is  difficult  to  convey  the  needs  that  non-­‐traditional  REU  students  have  to  mentors.  For  example,  mentees  with  families  or  jobs.  If  possible,  try  to  align  them  with  mentors  who  have  similar  backgrounds.  

 What  can  the  PAN  REU  community  do  to  help  in  this  task?  

10  

● Provide  links  to  mentoring  support  resources  on  a  Pan  REU  website  ○ Entering  Mentoring  (http://researchmentortraining.wceruw.org  )  ○ Center  for  the  Improvement  of  Mentored  Experiences  in  Research  (http://cimerproject.org/#/)  ○ Center  for  the  Integration  of  Research  Teaching  and  Learning  (CIRTL,  http://www.cirtl.net)  ○ National  Research  Mentoring  Network  (NRMN,  https://nrmnet.net  )ex  ○ Iowa  Center  for  Research  by  Undergraduates  (  http://www.uiowa.edu/icru/mentored-­‐research)    ○ APS  Bridge  Program  (http://www.apsbridgeprogram.org/conferences/summer13/pfund.pdf  )  

Program  Evaluation  The  discussion  prompts  were:  How  do  you  use  program  evaluation  to  evaluate  and  improve  your  program?  What  tools  have  you  found  to  be  especially  effective?  Have  you  developed  special  tools  or  methods  that  you  are  willing  to  share?  What  challenges  do  you  face  in  program  evaluation?  What  can  the  PAN  REU  community  do  to  help  in  this  task?    Currently  used  methods  

• Focus  groups  • Assessment  for  program  improvement  • Assessing  with  surveys  vs  demonstration  (presentations,  posters,  etc.)  • Using  survey  monkey,  many  programs  use  Pre  &  Post  surveys  to  assess  gains  • Midterm  mentee  surveys  are  used  to  monitor  how  things  are  going  and  to  provide  mid-­‐course  

corrections.  (social  activities,  dorms,  training  workshops,  etc.)  • Survey  the  mentors  midway  (face-­‐to-­‐face)  and  at  end  of  the  summer  (online).    • Brown-­‐bag  lunches  for  research  mentors  from  across  a  variety  of  REUs  on  campus  to  talk  about  issues,  

sharing  expertise,  etc.  Occurs  twice  during  the  summer.  • After  each  enrichment  lecture,  quickly  survey  what  worked  and  what  did  not.  • Use  long-­‐term  tracking  to  determine  impact  on  students.  • Create  a  Linked  In  profile,  Facebook  page  or  other  social  media  group  on  the  first  day  of  the  REU  for  

long  term  tracking  • Tools:  URSSA,  SURE,  Colorado  State,  UNC  Charlotte-­‐Audrey  Rorrer,  WICER  

   Issues,  Concerns,  Challenges  

• Costs  of  hiring  evaluator  • Not  sure  what  we  should  be  assessing  • Effectiveness  –  are  we  meeting  program  goals?  • Multiple  tools  may  be  necessary  because  some  goals  are  met  over  the  summer  (lab  skills,  etc.),  others  

are  more  long  term  (retention  in  school,  participation  in  STEM  careers)  • There  is  need  for  a  common  tool  that  would  assist  PIs  because  most  don’t  have  expertise  or  funds  for  

external  assessment,  and  it  would  also  help  reviewers  with  proposal  evaluation  knowing  that  we  are  checking  the  right  things  

• Concerns  about  the  effectiveness  of  a  common  assessment  tool  given  the  complexity  and  diversity  of  outcomes  for  REUs.    Perhaps  a  database  of  constructs,  items,  and  then  data  from  lots  of  REUs.  More  like  a  menu  people  can  select  from…  add  and  test  their  own  etc.      

• There  are  no  good  publication  venues  for  REU  evaluations—the  results  are  not  being  accepted  at  top  journals.    (Need  IRB  approval  to  use  human  subjects  data  on  REU  evaluations.)  

• How  do  we  share  results  of  education  research  based  on  REU  programs?  

11  

 Suggestions  

• A  common  instrument  with  a  menu.  For  example:  A  set  of  ~5  questions  that  NSF  requires  to  be  included  and  other  questions  that  PIs  can  choose  from  (a  question  bank),  and  an  additional  function  to  create  custom  designed  questions  to  accommodate  the  specific  needs  of  each  site  

• Using  centralized  survey  resource  (eg  URSSA)  is  of  benefit  to  NSF  for  evaluating  overall  success  • Some  directorates  have  a  common  tool  (BIO,  CISE)  –  compare  these  existing  tools  to  find  overlap  and  

broader  application  across  the  directorates  • Create  a  NSF  database  as  a  way  to  learn  what  elements  of  an  REU  site  other  PIs  are  investigating  in  

depth…  e.g.  Mentor/mentee  relationships,  recruitment,  selection,  etc.      • PIs  would  like  to  know  what  information  and  data  is  useful  to  NSF  so  that  we  can  be  sure  that  all  REUs  

are  collecting  this  data  (consistently)  at  a  minimum.        • PIs  would  like  clear  expectations  from  NSF  expect  in  terms  of  evaluative  information.  Questions  that  

need  to  be  answered  should  be  specified.  • There  could  be  a  line  item  in  the  budget  for  evaluation  costs  

Participant  Tracking  In  this  discussion,  participants  shared  the  methods  that  they  used  to  track  REU  participants  after  their  summer  experience,  the  types  of  information  they  collect,  and  the  challenges  they  face.  Over  the  course  of  this  discussion,  a  number  of  other  issues  became  apparent  and  were  discussed.    The  discussion  prompts  were:  How  do  you  measure  the  long-­‐term  impact  of  your  program?  How  do  you  track  your  participants  after  they  leave  the  program?  In  each  case,  what  exactly  do  you  measure?    What  tools  do  you  use  for  this  task?  What  challenges  do  you  face  in  long-­‐term  tracking?  What  can  the  PAN  REU  community  do  to  help  in  this  task?    Tracking  Approaches    

• e-­‐mailing  surveys  to  students  • inviting  students  to  participate  in  Facebook  

groups  • requesting  that  all  participants  create  a  LinkedIn  account  and  then  following  their  professional  progress  

there  • sending  out  informal,  unstructured  questions  by  e-­‐mail  (“hey,  what’s  up?”);    • sending  out  more  structured  questions  by  e-­‐mail  • pre-­‐  and  post-­‐summer  experiences  surveys  and  focus  groups  • informal  or  formal  reunions  after  the  summer  experience.  

Tracking  was  performed  by  faculty,  staff,  or  by  external  evaluators.  One  issue  identified  by  discussants  is  that  plans  for  tracking  need  to  be  put  in  place  early  to  ensure  continuity  of  data  collection.  In  particular,  because  students  usually  register  for  REU  programs  with  their  undergraduate  e-­‐mail  address,  which  typically  become  invalid  when  they  graduate,  a  permanent  e-­‐mail  address  or  other  modes  of  contact  may  be  necessary.  Most  

Figure  2.  REU  PIs  reported  on  breakout  group  discussions.  

12  

discussants  reported  that  they  were  able  to  track  a  relatively  high  (70-­‐90%)  proportion  of  participants.  It  was  noted  that  a  more  personal  connection  to  the  student  participants  results  in  a  better  rate  of  response.    From  this  short  discussion  of  methods  of  tracking  REU  participants,  two  larger  ideas  became  immediately  apparent.  The  first  recognized  that  the  mechanisms  for  tracking  and  evaluation  are  inextricably  linked.  Different  methods  of  tracking  reflect  different  modes  of  evaluation:  for  example,  tracking  students  via  LinkedIn  provides  information  on  their  professional  activities,  while  tracking  students  via  email  surveys  allows  for  the  collection  of  a  wider  range  of  data.    The  second  idea  was  that  individual  REU  sites  use  different  tracking  and  evaluation  methods,  which  reflect  their  own  mission  and  values,  and  it’s  unclear  how  these  REU-­‐specific  approaches  provide  information  to  the  NSF  and  to  political  decision  makers.    Different  levels  of  tracking  and  evaluation  were  identified.  A  primary  issue  was  collecting  information  about  REU  participants  to  be  made  available  to  Congress  to  demonstrate  positive  outcomes.  This  data  is  likely  to  focus  on  the  professional  progress  of  REU  participants  (such  as  engagement  in  further  STEM  training  or  a  career  in  a  STEM  field),  with  a  timeline  of  perhaps  five  years.  Based  on  the  comments  of  Assistant  Director  of  Engineering  and  Human  Resources,  Joan  Ferrini-­‐Mundy,  who  spoke  at  the  workshop,  the  primary  interest  of  the  NSF  is  that  students  have  positive  experiences  during  the  program;  tracking  for  this  purpose  may  then  focus  closely  on  the  summer  experience  itself,  collecting  information  immediately  before  and  after  the  program.  The  evaluation  may  address  the  development  of  knowledge,  skills,  and  self-­‐efficacy  in  the  research  field,  using  a  mixed-­‐methods  approach.  In  particular,  open-­‐ended  questions  provide  the  opportunity  to  identify  important  elements  of  the  REU  participant  experience  that  were  not  addressed  by  the  quantitative  questions.  Finally,  individual  REU  sites  have  specific  interests  and  missions  that  bear  on  tracking.  For  example,  Ian  Billick,  from  the  Rocky  Mountain  Biological  Laboratory,  reported  that  they  track  their  REU  participants  indefinitely.  This  is  in  line  with  their  mission  to  support  scientists  over  their  entire  career,  as  well  as  to  foster  a  coherent  community  of  trainees  and  research  professionals.  Conversely,  other  discussants  noted  that,  after  the  end  of  the  grant  funding  period,  REU  sites  may  still  be  expected  to  collect  information  despite  not  having  any  financial  support  to  do  so.  An  example  of  REU-­‐specific  tracking  would  be  that  programs  that  focus  on  participants  from  community  colleges  may  track  the  proportion  of  students  who  transfer  to  universities.  Other  examples  of  REU-­‐specific  evaluations  included  the  development  of  self-­‐identity  as  a  scientist,  or  the  impact  of  a  research  experience  on  how  participants  engaged  in  the  remainder  of  their  academic  program.      The  key  request  for  follow-­‐up  that  emerged  from  this  group  was  for  clarity  around  evaluation  strategies  for  the  NSF  and  for  reporting  to  Congress,  with  the  understanding  that  these  strategies  would  shape  tracking  methods  and  duration  (for  example,  an  NSF-­‐wide  REU  tracking  program),  as  well  as  appropriate  funding  strategies.  

REU  Leadership  Group  The  BIO,  CHE,  GEO,  and  CISE  Directorates  have  a  leadership  group.  PHY  is  in  the  early  stages  of  developing  a  group  and  AST  is  small  enough  that  the  whole  set  of  PIs  functions  as  a  leadership  group  (n=20).    Materials  science  has  no  formal  leadership  group,  however  they  do  have  a  list  serve  and  run  a  web  site.  SBE  (Social,  Behavioral  and  Economic  Sciences)-­‐does  not  have  a  leadership  group.  Mathematics-­‐has  an  “informal”  leadership  group  Biology-­‐has  a  formal  leadership  group,  very  structured  Chemistry-­‐has  a  formal  leadership  group  Geoscience-­‐has  a  “loose”  leadership  group;  CISE  (Computer  &  Information  Science  and  Engineering)  –has  a  “loose”  leadership  group.  Some  institutions  have  a  leadership  group  for  the  REUs  at  their  institutions  from  different  disciplines,  and  share  a  common  application  (e.g.  in  CISE).  Others  collaborate  between  neighboring  institutions.  The  directorate  program  officer  frequently  nominated  the  leadership  group  members.      The  need  for  and  structure  of  an  REU  leadership  group  in  any  given  division  is  related  to  the  size  of  the  division.  It  is  done  both  with  and  without  external  funding.    The  nature  of  the  leadership  group,  structured  or  organic,  

13  

correlates  to  the  number  of  REU  sites  within  the  division.  BIO,  for  example  has  over  135  sites  and  therefore  has  to  be  more  structured.    If  there  are  distinct  “leaders”  within  a  directorate,  how  should  they  be  selected?    (This  currently  varies.)      The  group  discussed  tracking  REU  participants,  collaborating  with  and  supporting  with  faculty  at  MSIs,  and  creating  a  venue  where  REU  PIs  to  publish  on  their  practices.    Another  function  identified  of  a  leadership  group  is  to  advocate  for  research  and  service  credit  for  running  REU  programs.        Several  advantages  of  leadership  groups  were  identified:    

• a  good  means  by  which  new  PI’s  can  find  more  experienced  PI’s  for  mentorship  and  advice    • with  some  groups  new  PI’s  are  able  to  access  resources  from  the  group;  e.g.  drafts  for  applications,  

letters,  etc.    • supports  the  creation  of  new  tools  such  as  common  applications,  common  evaluation  surveys  and  more  • being  informed  about  NSF  priorities  which  change  over  time  • communication  occurs  by  list  serves;  with  list  serv  it  is  possible  for  PI’s  to  exchange  ideas,  give  advice  in  

real  time  as  issues  arise  and  prompt  community  conversations    • other  methods  of  communication  include  websites,  Facebook  -­‐PI’s  have  also  received  advice  for  

renewals  from  persons  they  have  met  at  leadership  group  meetings  • in  cases  where  more  experiences  PI’s  are  asked  to  share  best  practices,  the  more  experienced  PI’s  are    • forced  reflect  on  why  they  “do  what  they  do”  • overall  leadership  groups  are  sources  of  support  for  PI’s    

 Suggestions  for  consideration:    

• implementation  of  a  leadership  group  for  SBE,  especially  because  this  area  is  so  interdisciplinary    • a  need  for  a  PAN  REU  leadership  group  (across  directorates)  and  provide  one  place,  ex.  a  website,  where  

all  PI’s  can  find  resources  that  are  common  to  all  sites  as  well  as  resources  that  are  categorized  by  discipline.    

- a  website  that  provides  information  about  current  interests  of  leadership  groups.    - potential  need  for  a  MSI-­‐focused  leadership  sub-­‐groups  as  well  as  PUI  and  CC  subgroups  - creates  synergistic  leadership  when  institutions  have  weaker  sponsored  programs  offices  

leadership  group    • conserving  resources  as  the  make-­‐up  of  leadership  group  changes,  e.g.  consistency  in  websites  • sometimes  more  experienced  PI’s  do  not  have  an  opportunity  to  address  issues  that  they  face  • consider  the  source  of  funding  for  the  leadership  group;  would  like  to  balance  this  demand  with  the  

ability  to  fund  more  REU  programs    • ensure  clarity  of  communication  about  leadership  groups  to  new  PIs  • preserve  best  practices  for  PI  generations  to  come.  Sometimes  there  is  a  loss  of  information/resources  

when  membership  of  the  Leadership  group  changes.  

Career  Development  for  Students    Participants  shared  several  career  development  activities  for  REU  students  that  are  needed  in  entering  and  succeeding  in  education  and  the  workforce.  These  activities  develop  skills  in  presenting,  writing,  preparing  application  materials,  leadership,  entrepreneurship,  mentoring,  innovation,  interdisciplinary  thinking,  and  critical  thinking.    Participants  described  exercises  in  extemporaneous  speaking,  elevator  speeches,  résumé-­‐writing,  mini-­‐courses,  equipment  training,  and  team  building.    

14  

It  was  agreed  that  students  need  greater  exposure  to  career  options  in  the  sciences  in  different  areas  of  academia,  industry,  and  the  private  sector.    Several  ideas  were  proposed:    

• Ask  all  visitors  and  instructors  in  the  REU  program  to  talk  about  their  own  career  path  and  give  career  advice,  including  those  giving  science  talks  and  those  leading  training  in  communication,  technical  skills,  trips  or  workshops;  

• Make  visits  to  local  labs,  natural  history  museums,  consulting  or  non-­‐profit  firms,  job  airs,  and  graduate  schools,  and  where  possible,  ask  hosts  to  describe  their  career  paths;  

• Start  developing  connections  and  a  webpage  with  alumni  profiles  describing  their  occupations  in  the  workforce  and  promoting  mentoring  of  the  REU  students  by  providing  emails  for  contact;  

• For  programs  with  veterans  not  thinking  about  academia,  who  need  to  survive  college  and  get  a  job,  help  them  with  confidence  first,  then  skill  building,  entrepreneurship  training,  and  have  them  write  small  business  plan  around  their  project.    Encourage  research  mentors  to  talk  about  careers  and  family  life,  not  just  research,  and  students  to  ask  questions  on  these  topics.  

• Encourage  career  planning  using  individual  development  plans  (e.g.  myidp.sciencecareers.org)  with  exercises  to  help  students  examine  their  own  skills,  interests,  and  values.  These  tools  can  help  students  to  identify  which  science  career  paths  might  suit  them  and  for  setting  goals  that  they  can  revisit.  

 Research  mentors  may  push  back  on  their  mentee  spending  time  on  career  development  activities  and  away  from  doing  research.  However,  during  pre-­‐program  preparation,  the  REU  coordinator  or  PI  should  explain  their  importance  and  require  that  mentors  support  this  program  element.    Requiring  students  to  attend  all  of  or  a  certain  percentage  of  these  activities  is  effective.    The  group  recommended  that  the  community  be  asked  to  share  ideas,  links,  and  other  resources  so  that  they  may  be  included  on  the  pan-­‐REU  website.  It  was  suggested  that  a  subcommittee  on  this  topic  could  help  to  build  up  resources.  One  participant  suggested  obtaining  funds  for  student  leaders  to  develop  resources  on  career  development  for  the  REU  community,  perhaps  through  an  REU  supplement.  REU  alumni  were  suggested  as  being  a  valuable  resource,  as  mentors,  people  providing  examples  of  career  paths,  and  as  guides  to  the  undergraduates  in  developing  resources.    Such  a  program  would  take  some  

Recruiting  and  Selecting  to  Broaden  Participation  Participants  were  asked  to  describe  their  recruiting  activities  with  the  following  discussion  prompts:  Does  your  program  certain  underrepresented  groups,  and  if  so  how  and  why?  What  challenges  do  you  face  in  broadening  participation  in  your  program?  What  can  the  PAN  REU  community  do  to  help  in  this  task?    This  breakout  session  focused  on  defining  continuing  challenges  and  identifying  successes,  failures  and  best  practices  to  address  such  challenges.      Remaining  challenges:  

• Specific  underrepresented  groups  (i.e.,  AA/Black  Males)  are  consistently  the  least  represented  at  the  earliest  stages  of  the  application  process  and  may  require  unique  recruitment  strategies.  

•  Indicators  of  both  potential  (preparation,  and  for  persistence)  and  definitions  of  success  are  quite  variable  across  divisions  and  individual  sites  (i.e.,  some  are  GPA-­‐based,  some  are  project  specific,  etc.)  and  may  inadvertently  discourage  applications  from  target  groups.  

•  International  sites  introduce  new  challenges  beyond  cultural  adjustment  that  can  influence  recruitment  and  success.  

15  

• There  is  a  distinct  lack  of  confidence,  fit,  and  self-­‐efficacy  in  applicants  that  is  often  reinforced  by  a  lack  of  home  faculty  encouragement  to  apply  such  programs.  

• The  pool  of  applicants  is  not  simply  a  function  of  gender/ethnic/racial,  or  other  underrepresented  designation,  but  also  often  is  confounded  with  important  life  circumstances  that  limit  perceived  ability  to  participate  in  REUs  (i.e.,  travel  limitations,  fiscal  or  family  responsibilities,  nonacademic  career  goals).  

• REUs  as  a  general  program  still  have  a  low  profile  in  the  UG  community,  and  there  remains  a  real  lack  of  awareness  of  the  very  existence  of  REUs.  

   Ways  the  community  may  address  these  challenges:  

• Common  methods  are  still  moderately  effective  and  should  be  maintained  or  further  expanded.  These  include:  A  social  media  and  online  presence,  outreach  and  collaboration  across  programs  and  institutions,  alumni  development,  utilizing  individual  networks  and  professional  connections  to  spread  the  word.  

•  Recruit  in  cohorts,  rather  than  individually.  Further,  extend  cohort  building  across  REUs  at  same  or  nearby  institutions  and  create  larger  UG  research  network.  

•  Know  your  pool!  To  the  extent  possible  –  know  your  population  and  the  networks  that  can  best  access  them  (i.e.,  Veterans  and  ROTC  or  VA),  and  the  limitations  that  come  from  that.  

o Continuously  monitor  needs  of  participants,  they  are  ever  changing  –  evolving  methods,  constant  evaluation  of  success  and  efficacy  can  produce  high  or  increased  recruitment  of  target  groups.  

• More  targeted  one-­‐on-­‐one,  face-­‐to-­‐face,  and  skype  recruiting  methods  have  been  successful  lately  (i.e.,  going  to  each  UG  poster  at  national  conferences  and  personalizing  invitations  to  apply).  

•  Build  REAL,  and  sustainable,  relationships  with  MSIs  rather  than  simply  disseminate  information  about  program.  These  could  be  in  the  form  of  collaborations,  shared  PI  roles,  etc.  

• Utilize  LSAMP  and  McNair  networks  to  disseminate  and  establish  continuous  pipeline.  •  There  is  a  strong  desire/need  for  discipline  specific  centralized  databases  that  have  contact  info  for  

departments/UG  research  offices  at  MSI  and  two  year  institutions.  It  seems  that  programs  each  (re)create  these  lists  individually,  and  there  is  too  much  reinventing  of  the  wheel.  

• No  Repeats  rule  –  with  exceptions,  so  that  opportunities  are  not  monopolized  by  a  small  group  of  students  who  are  advantaged  by  prior  experience.  

• News  Blasts  and  mailings  from  NSF  about  the  programs  to  raise  profile  of  REU  program  broadly.  

Common  Application  Deadline  These  breakout  groups  were  asked  to  discuss  common  aspects  of  administering  an  REU  program  in  your  discipline.  (e.g.  application,  deadlines  and  acceptance  dates,  and  evaluation).  The  discussion  prompts  were:  How  were  the  common  aspects  created  and  how  are  they  maintained?  What  are  the  challenges  in  developing  and  maintaining  this  common  administration?  Which  common  tools  should  be  voluntary,  and  which  should  be  required?    What  can  the  PAN  REU  community  do  to  help  in  developing  common  tools?    Common  application  deadline  and  acceptance  date  

• The  astronomy  community  uses  a  common  acceptance  date.  It  is  a  smaller  community  and  there  is  a  gentleperson’s  agreement  that  students  are  accepted  March  1.    If  programs  need  to  accept  later,  (e.g.  they  are  funded  late)  then  it  is  fine  to  delay.    Students  have  a  week  to  respond.    The  process  allows  students  to  know  whether  their  top  program  has  accepted  them  before  having  to  commit.    It  also  reduces  problems  for  programs  by  reducing  the  number  of  students  backing  out  of  commitments.  

16  

• One  participant  expressed  concern  that  the  way  they  process  applications  and  accept  students  does  not  lend  itself  to  accepting  students  at  a  single  time.    He  asked  about  using  a  common  response  period  instead.  

• There  was  a  general  sense  that  maybe  each  leadership  group  could  suggest  a  date  for  acceptance/response  and  then  individual  programs  could  decide  whether  to  use  the  common  date.  

• There  was  a  strong  sense  that  programs  should  not  be  required  to  use  a  common  application.    There  were  concerns  about  a  common  application  creating  barriers  for  programs  using  program-­‐specific  questions,  the  creation  of  barriers  for  underprivileged  students  to  applying,  and  the  degradation  of  the  ability  of  programs  in  identifying  students  who  are  a  good  fit.  

• It  was  noted  that  a  common  application  tool  might  facilitate  an  accurate  estimate  of  the  total  number  of  unique  applicants  to  the  REU  program  as  well  as  reduce  the  number  of  reference  letters  being  written  by  faculty  as  well  as  reduce  the  time  needed  to  apply  to  multiple  programs.  

• There  was  a  suggestion  that  programs  could  ask  participating  students  how  many  programs  they  had  applied  to  in  order  to  estimate  the  total  number  of  unique  applicants.  

Short  Talks    

GEO:  Val  Sloan  and  David  Fields  The  history  of  how  the  GEO  REU  network  developed  was  described  starting  with  the  first  GEO  REU  PI  workshop  in  2011  that  was  initiated  by  the  NSF.    A  significant  impact  of  that  meeting  was  the  face-­‐to-­‐face  connecting  by  REU  PIs,  which  allowed  continued  discussion  and  sharing  of  resources  after  the  workshop.    One  outcome  of  that  meeting  was  the  development  of  the  GEO  REU  email  listserv,  which  has  been  used  for  discussion  between  PIs  about  how  to  design  specific  programmatic  elements  or  address  issues,  for  timely  reminders  and  resource  support  throughout  the  annual  REU  cycle,  and  for  the  planning  and  announcements  of  events  or  opportunities.    Oral  and  poster  sessions  on  running  an  REU  were  initiated  at  the  annual  meeting  of  the  American  Geophysical  Union  in  2011  and  have  been  organized  each  year  since.    In  2014  new  REU  PIs  were  introduced  to  the  community  at  the  second  REU  PI,  and  a  workshop  outcome  was  a  GEO  REU  webpage.      

CHE:  Linette  Watkins  The  chemistry  REU  leadership  group  (LG)  grew  out  of  a  principal  investigator  (PI)  meeting  that  was  held  in  2001.  The  LG  is  comprised  of  REU  site  directors  and  is  currently  funded  by  a  special  grant  from  the  NSF  (CHE#1258759).  The  group  seeks  to  advance  the  REU  program  by  running  workshops  and  symposia  for  REU  PIs.  Through  these  workshops  and  symposia,  the  LG  learns  what  are  the  most  pressing  issues  facing  the  community  and  how  to  possibly  addresses  these  issues.  In  addition,  the  group  serves  as  a  liaison  between  the  broader  chemistry  REU  community  and  the  NSF.  Currently,  the  issues  of  greatest  interest  to  the  chemistry  REU  community  are:  strategies  for  broadening  participation,  tracking  REU  participants,  mentor  training  and  site  evaluation  /  assessment.    

BIO:  Alan  Berkowitz    The  Biology  Leadership  Council  (BLC)  was  initiated  in  2006  by  Sally  O’Connor,  when  she  brought  together  principal  investigators  from  ten  institutions  representing  a  diverse  array  of  programs  and  topics.  The  members  now  have  staggered  terms.  The  Division  of  Biological  Infrastructure  provides  grants  to  support  meetings  and  specific  activities  that  have  led  to  the  development  of  several  shared  

Fig.  3.    After  four  speakers  described  their  disciplines’  REU  Leadership  Groups  (GEO,  CHE,  BIO,  AST),  the  workshop  participants  discussed  ways  of  running  such  a  group.

17  

resources.  One  REU  site  director  is  supported  to  attend  each  meeting  through  their  individual  REU  site  grants.  The  BLC  organizes  a  meeting  every  two  to  three  years  to  identify,  plan  and  develop  cross-­‐cutting  initiatives  for  the  biology  REU  sites.  Among  the  major  outcomes  of  these  meetings  are  the  common  assessment  tool  (Undergraduate  Research  Student  Self  Assessment  (URSSA)),  travel  awards  for  students  and  a  discontinued  Just-­‐in-­‐Time  reviewing  process.  These  meetings  were  also  instrumental  in  the  establishment  of  a  one-­‐stop  shop  for  the  BIO  REU  program,  as  well  as  the  ethics  training  program  and  the  responsible  conduct  of  research  supplements.  Another  key  benefit  of  the  work  done  by  the  BLS  is  sharing  resources  on  how  to  recruit  and  support  students  from  underrepresented  groups.  

AST:  Kathy  Eastwood  The  Division  of  Astronomical  Sciences  Leadership  Group  formed  organically.  All  REU  Site  directors  are  in  a  Google  email  group  for  communications  and  invited  to  a  meeting  about  every  5  years.    The  current  list  of  cooperating  sites  (essentially  all  of  AST’s  REU  portfolio)  includes:  

● American  Museum  of  Natural  History  and  the  City  University  of  New  York  ● Arecibo  Observatory  ● Boston  University  ● Cornell  University  ● Keck  Northeast  Astronomy  Consortium  ● Louisiana  State  University  ● Maria  Mitchell  Observatory  ● National  Optical  Astronomy  Observatory  ● National  Radio  Astronomy  Observatory  ● National  Solar  Observatory  ● Northern  Arizona  University  and  Lowell  Observatory  ● Northwestern  University/CIERA  ● Rutgers,  The  State  University  of  New  Jersey  ● SETI  Institute  ● Smithsonian  Astrophysical  Observatory  ● Texas  A&M  University  ● Texas  Christian  University  ● University  of  Wisconsin-­‐Madison  ● University  of  Wyoming  

 Dr.  Eastwood  described  two  currently  active  collective  initiatives  in  AST.      

• Coordinated  recruiting  at  Minority-­‐Serving  Institutions.    Clearly  a  difficult  problem  to  solve,  a  new  idea  by  Andrew  Baker  (Rutgers  REU  Site  director)  has  just  been  implemented  involving  face-­‐to-­‐face  visits.    An  REU  director  gives  talks  at  MSIs,  with  a  special  communication  about  all  AST  REU  Sites.    Students  are  engaged  in  person,  and  faculty  relationships  are  built.    The  REU  director  then  shares  student  information  with  other  AST  REU  Site  directors,  creating  a  larger  electronic  recruiting  network.    Although  it’s  too  early  to  tell  if  this  strategy  will  be  effective  long-­‐term,  so  far  it  has  directly  resulted  in  3  REU  acceptances  in  AST.  

• A  common  date  for  first  offers  of  March  1,  with  an  earliest  acceptance  deadline  of  March  8.    The  application  dates  may  vary,  and  it’s  not  always  possible  for  every  REU  Site  to  participate  every  year,  but  each  student  has  at  least  a  full  week  to  decide,  and  the  results  are  “highly  beneficial  to  students.  Physics  and  DMR:  Please  join  us!    We’re  all  recruiting  the  same  students.    Email:    ast-­‐reu-­‐site-­‐directors@googlegroups.com.”  

 

18  

Common  REU  Assessment  &  Evaluation  Tool  

Common  Assessment  &  Evaluation  Tool  Disciplinary  Group  Discussions    A  presentation  outlining  the  development  of  a  new  NSF  REU  common  assessment  tool  being  developed  at  the  University  of  Wisconsin  –  Madison  was  given,  and  this  was  followed  by  breakout  sessions  at  which  each  disciplinary  division  considered  the  following  questions:  

• What  should  be  modified,  deleted,  or  added  to  make  the  surveys  applicable  to  your  discipline?  • What  challenges/questions  might  arise  from  the  PIs  in  your  discipline  when  asked  about  adopting  a  

common  assessment  and  evaluation  tool?  • What  would  you  like  the  tool  to  do  that  it  currently  cannot  do?  

 AST  &  PHY:  The  current  survey  would  be  inappropriate  for  physics  and  astronomy  programs,  then  considered  whether  a  common  assessment  with  a  different  set  of  questions  could  have  some  value.  The  conclusion  was  that  it  might  have  value  but  the  set  of  common,  validated  questions  should  be  short  (less  than  half  a  dozen  or  so).  We  were  also  very  worried  that  a  common  set  of  assessment  questions  would  have  the  danger  of  inhibiting  flexibility  and  creativity  in  future  programs  if  people  tailored  their  program  to  match  the  assessments.    BIO:  The  BIO  REU  sites  are  already  using  a  common  assessment  tool,  but  the  functionality  of  this  tool  has  been  a  problem.  It  required  the  REU  leadership  group  to  develop  a  detailed  set  of  instructions  for  individual  PIs  and  also  is  not  able  to  aggregate  data  across  REU  programs.  Development  of  the  new  tool  was  meant  to  address  these  shortcomings.  The  group  also  discussed  the  common  questions  that  each  site  is  required  to  use  and  agreed  that  the  BIO  PI  community  at  large  needs  to  reevaluate  these  questions  at  the  next  PI  meeting  in  order  to  prioritize  and  shorten  this  list.  There  was  limited  discussion  of  the  parallel  mentor  survey  in  the  new  tool,  since  the  NSF  will  not  require  sites  to  use  it  (it  is  an  optional  feature).  The  group  agreed  that  continued  ability  to  design  program  specific  customized  questions  and  to  download  a  formatted  report  was  important  for  ease  of  use  for  PIs.    CHE:    The  CHE  group  discussed  the  implementation  of  a  common  assessment  tool.  Currently  CHE  REU  sites  use  a  variety  of  tools  including  SURE  III  and  SALG,  but  most  reported  in  2015  that  sites  use  tools  that  they  developed  themselves.  The  CHE  representatives  echoed  what  was  discussed  at  the  CHE  PI  meeting  in  2015.  First,  it  was  recognized  that  significant  resources  are  required  for  a  robust  common  assessment  and  secondly,  most  of  the  available,  open  access  tools  do  not  assess  the  programmatic  aspects  that  most  interest  the  CHE  REU  community.  There  is  also  the  concern  as  to  how  the  data  will  be  used.  Site  directors  are  interested  in  robust,  validated  assessment  tools  that  will  make  their  sites  better.  Representatives  felt  strongly  that  some  assessment  should  be  shared  with  NSF  Program  Officers  and  the  greater  REU  community,  but  that  some  of  the  assessment  data  should  belong  to  the  specific  REU  site  director(s)  as  they  endeavor  to  strengthen  their  programs.      CISE:  The  CISE  group  already  has  a  common  assessment  tool  that  up  to  75  of  their  sites  have  been  using  since  2010  (http://reu.uncc.edu/toolkit/la-­‐carte-­‐survey).  They  would  consider  adopting  a  common  tool  that  would  be  used  beyond  CISE.  They  would  require  flexibility  in  the  tool  to  be  able  to  add  their  own  questions  to  the  same  core  survey.  They  discussed  “survey  fatigue”  for  students  (and  faculty)  and  decided  that  any  common  questions  and  instruments  should  be  incorporated  into  site-­‐specific  surveys  to  avoid  deploying  multiple  surveys.  They  noted  that  having  a  common  survey  tool  would  reduce  their  evaluation  costs  by  eliminating  the  need  to  hire  an  evaluator.  However,  the  qualitative  feedback  they  receive  from  the  evaluator  could  not  be  replaced  by  a  common  survey.    EHR  and  SBE:  A  common  assessment  would  provide  clarity  to  new  and  prospective  PIs  about  what  types  of  impact  REU  sites  are  expected  to  have,  and  this  assessment  gets  at  some  of  the  questions  that  the  NSF  and  

19  

Congress  might  need  to  evaluate  the  REU  program.  However,  until  there  are  clearly-­‐defined  goals  that  are  shared  by  all  REUs,  creating  an  NSF-­‐wide  assessment  tool  is  premature.  Several  other  specific  issues  were  identified.  As  representatives  of  EHR  and  SBE,  we  immediately  noted  that  this  assessment  uses  language  that  is  not  generalizable  to  all  divisions.  Next,  any  assessment  tool  should  include  open-­‐ended  questions  in  addition  to  quantitative  methods;  besides  providing  a  broader,  more  nuanced  picture,  qualitative  responses  are  useful  to  illuminate  elements  of  the  student  experience  that  were  not  captured  in  the  quantitative  questions  (particularly  as  the  program  and  the  participants  change  and  evolve).  Finally,  while  the  focus  on  the  development  of  skills,  knowledge  and  self-­‐efficacy  is  laudable,  this  instrument  does  not  capture  other  elements  that  are  developed  in  REU  programs,  including  professional  skills,  networks,  and  the  ability  to  communicate  and  disseminate  results.    ENG:  The  engineering  group  would  like  the  tool  to  be  shorter  than  it  currently  is  and  would  prefer  to  have  a  bank  of  validated  questions  from  which  PIs  could  choose.  They  would  also  like  the  tool  to  incorporate  tracking  functions,  including  collection  of  data  about  retention,  graduation  rates,  publications,  conference  attendance,  and  careers/jobs.  They  noted  that  an  NSF  evaluator  should  administer  the  survey  that  addresses  NSF  goals,  and  that  this  should  free  up  for  use  on  other  program  components.  Engineering  PIs  would  like  to  have  ownership  of  the  data  collected  using  the  tool  for  their  own  research  and  dissemination  purposes.  They  indicated  that  they  would  like  to  have  common  data  from  their  program  applicants  as  well  as  their  program  participants  and  suggested  that  a  common  application  be  considered.        GEO:    Issues  around  access  to  aggregated  data  collected  from  a  commonly  used  assessment  tool  were  discussed,  and  IRB  limitations  on  publishing  data  collected  from  a  range  of  institutions.    Specifically,  if  an  IRB  is  required  from  all  institutions  whose  data  are  to  be  included  in  an  aggregated  data  set  intended  for  a  publication,  the  effort  involved  in  obtaining  those  IRBs  may  be  cost/time-­‐prohibitive.    If  aggregated  data  are  not  publishable  for  IRB  reasons,  the  interest  in  conducting  a  common  evaluation  would  drop.    Having  said  that,  participants  saw  the  value  in  having  a  core  set  of  questions:  (1)  to  make  evaluation  easier  for  GEO  REU  sites,  and  (2)  to  obtain  data  from  GEO  REUs.    The  group  identified  concerns  it  had  about  using  the  BIO  group’s  new  assessment  tool,  and  follow-­‐up  discussion  with  BIO  REU  representatives  indicated  that  they  may  be  changing  their  pool  of  questions.    In  this  discussion,  participants  expressed  an  interest  in  obtaining  information  on  the  student  application  pool,  including  the  number  of  unique  applicants,  the  demographics  of  who  applies  to  which  GEO  REU  programs  and  is  accepted  by  the  programs.    We  identified  the  value  of  having  a  unique  identifier  for  each  applicant,  and  for  the  purposes  of  discussion,  called  it  an  REU  ID  number  (or  REUID).    If  students  had  to  obtain  an  REUID  before  applying  to  any  REU,  and  in  doing  so,  entered  their  demographic  information,  we  would  be  able  to  characterize  the  application  behavior  of  students  across  GEO  REUs,  and  the  demographics  of  the  accepted  students.    Hypothetically,  this  REUID  could  be  tracked  longitudinally  for  those  people  who  continue  to  be  involved  in  NSF  activities  such  as  applying  to  the  Graduate  Research  Fellowship  or  submitting  proposals.    Participants  also  discussed  conducting  a  small  pilot  study  to  characterize  the  applicant  pool  and  behavior  across  three  GEO  REU  programs.  

Action  Planning  Breakout  Session  Summaries  The  Pan  Leadership  Group  identified  several  topics  that  emerged  over  the  course  of  the  workshop  that  seemed  to  be  of  greatest  importance  and  for  which  action  plans  could  be  generated.  Participants  were  asked  to  join  groups  that  would  develop  action  plans  for  topics  of  interest  to  them  and  the  discipline  they  represented.  Each  group  was  given  the  following  questions  to  guide  their  action  planning  process:  

• How  do  members  of  the  group  define  this  topic?  What  are  the  major  commonalities  amongst  the  different  divisions?  

• What  are  one  to  three  goals  that  the  group  considers  of  high  importance  pertaining  to  this  topic?  

20  

• Per  goal,  what  is  one  action  item?  Consider  aspects  such  as  implementation,  feasibility,  resources  and  timeframe.  

Developing  Collaborations  with  MSI’s    Action  Item  1:  Create  a  leadership  team  within  the  Pan  REU  to  collect  and  share  information  to  facilitate  these  collaborations.  The  leadership  team  will  be  chosen  from  existing  teams  from  each  NSF  divisions  where  they  exist.  Where  they  don’t  exist,  we  will  ask  for  interested  volunteers  using  mailing  lists  of  REU  site  directors.  Steve  Turley  will  collect  the  list  of  site  director  email  addresses.    Action  Item  2:  We  will  create  a  report  of  suggested  practices  for  establishing  effective  practices  for  collaborating  between  MSI  and  majority-­‐serving  institutions.  Camellia  will  create  a  draft  report  by  June  1,  2016.  We  will  then  circulate  a  report  among  our  group  and  produced  a  final  report  by  July  1,  2016  for  publication  on  the  Pan  REU  web  site.  Michael  will  apply  for  a  supplemental  grant  to  provide  support  for  Camellia  to  work  on  this.      Action  Item  3:  Organize  a  workshop  on  effective  collaborations  with  minority-­‐serving  institutions  in  conjunction  with  an  existing  national  meeting  such  as  Beta  Kappa  Chi,  SACNAS,  or  AISIS.  The  leadership  group  will  organize  the  first  activity  in  2017.  

Developing  Collaborations  with  Community  Colleges    Action  Item  1:  Each  PI  should  establish  connections  with  a  faculty  member  at  a  local  community  college  and  a  faculty  member  at  a  community  college  at  a  distant  campus.  Connections  with  regional  hubs  will  have  a  significant  impact.    Action  Item  2:  Empower  community  college  faculty  to  be  a  Co-­‐PI  or  Senior  personnel  in  the  renewal  and  involve  them  in  the  REU  programming  and  curriculum      Action  Item  3:  Broadening  participation  in  the  newly  established  leadership  group  (REU2N)  and  connect  with  the  Community  College  Undergraduate  Research  Initiative  (CCURI).  

Formalizing  the  Pan  REU  Community  and  Development  of  a  Pan  REU  Website    Goal  1:  Establish  online  community  and  resource  center  

• Make  website  and  resources  public  • Create  a  Listserv  or  Google  Group  that  is  member-­‐based  for  offline  discussion  among  PIs  and  

prospective  PIs    Action  Item  1:  Identify  people  and  institutions  to  write  an  RCN  proposal  to  coordinate  a  pan-­‐NSF  REU  network.    Action  Item  2:  For  the  website,  one  action  item  could  be  to  create  a  task  force  to  lead  on  this  topic,  and  a  chair  or  co-­‐chairs  of  the  task  force.  Names  suggested  as  representatives  from  each  Division  for  the  task  force  were  as  follows:  

• AST:  Kathy  Eastwood  • BIO:  Janet  will  identify  • CHEM:  Karen/Linette  will  identify    • CISE:  Dianxiang  Xu  suggested  Audrey  Rorrer  

21  

• GEO:  Val  Sloan,  Deanna  Erdner  • SBE:  Tim  Hawthorne  &  Rachel  Kallen  • M/PS:  representative  needed    • ENG:  representative  needed  

The  first  priority  of  this  task  force  would  be  to  work  with  those  writing  an  RCN  proposal  on  what  is  needed  for  defining  the  needs  for  a  pan  NSF  REU  web  page.    Action  Item  3:  Contact  NSF  staff  to  identify  who  might  consider  supporting  an  effort  to  develop  a  basic  website  in  the  short-­‐term,  if  that  is  a  priority.  Possible  contacts  at  NSF  include  Program  Director  Corby  Horbis,  the  Program  Officer  REU  working  group,  and  our  Division  Directors.    Action  Item  4:  Identify  an  organization  and  leaders  to  host  and  manage  an  email  listserv.  Topics  that  covered  on  the  website  could  include:  

• Links  to  discipline-­‐based  REU  leadership  group  web  pages  • Evidenced-­‐based  (best)  practices  • Mentoring    • Broadening  Participation  -­‐  recruitment  and  collaboration  • See  CHEM  topics  -­‐  comprehensive  array  of  tips  and  toolkits  (Karen/Linette)  • See  CISE  REU  website  tools  and  topics    • Communicating  with  Congress  and  public  • Collaborating  with  CUR  

 Goal  2:  Establish  a  PAN  NSF  REU  Leadership  Group    

• Establish  Formal  PAN  community  for  all  REUs  across  the  Divisions  • Long  Term  Advocacy  for  REU  agenda  more  broadly  

 Action  Item  1:  Merge/meet/coordinate  with  any  other  pan  task  forces  (REU2N,  Pan  MSI  LG)  

● Communicating  with  the  broader  REU  community    ● Requesting  resources  from  the  community  ● Timeframe:  Immediately,  and  continuing  _  goal  another  PAN  REU  meeting  in  a  few  years  ● Long-­‐term  sustainability  and  content  maintenance  

Informing  the  Development  of  Long-­‐Term  Tracking  Tools    Action  Item:  Though  there  were  reservations  about  the  creation  of  an  NSF  ID  for  REU  applicants,  there  was  general  support  for  a  pilot  project  in  a  limited  set  of  REU  sites,  perhaps  within  a  directorate  or  discipline.  In  general,  the  thought  is  the  student  would  interface  with  an  NSF  portal  to  enter  demographic  information  and  receive  an  ID,  perhaps  in  the  form  of  an  email  address.  This  ID  would  be  used  when  applying  to  individual  programs,  students  would  enter  this  ID  and  demographic  data  would  automatically  upload  to  applications,  saving  the  students  the  task  of  entering  the  same  demographic  data  in  multiple  applications.  Individual  sites  would  report  the  IDs  of  all  applicants  to  the  NSF,  who  would  ultimately  return  data  to  the  site  in  order  for  the  evaluate  aspects  of  their  sites,  for  example  the  success  of  their  recruiting  methods.  The  NSF  ID  could  be  presumably  used  when  participants  later  interfaced  with  the  NSF  (e.g.  through  applications  to  NSF  GRFP,  CAREER  grants,  and  other  later  individual  grants.)    Needed  Resources:    NSF  has  the  resources  at  their  end  to  create  the  portal  through  existing  contracts  and  the  Office  of  Assessment.  However,  PIs  would  encounter  additional  costs  for  programming  for  modification  of  their  

22  

applications.  Workshops  and  meetings  between  contractors  and  the  pilot  group  would  need  to  be  supported  for  implementation,  development,  improvement,  and  assessment  of  the  project.    Timescale:    Since  most  applications  are  opened  in  the  Fall,  it  is  probably  too  late  to  implement  this  for  the  next  application  cycle,  though  workshops  could  be  convened  for  further  discussion  such  that  the  tools  would  be  in  place  for  applications  for  programs  run  during  the  Summer  of  2018.    Additional  notes:  One  concern  about  this  plan  including  hesitation  at  broadening  the  scope  of  the  project  outside  what  the  America  Competes  Act  calls  for,  i.e.  tracking  applicants,  not  just  participants.  Also,  the  REU  community  is  hesitant  about  being  diverted  to  the  task  of  evaluating  the  impact  of  other  NSF  programs.  There  were  also  some  concerns  about  privacy.  Additionally,  some  people  thought  that  some  of  this  data  could  be  obtained  in  other  ways,  for  example,  by  asking  students  to  how  many  programs  they  applied  or  by  mining  the  existing  data  that  are  collected  by  PIs.  In  fact,  an  additional  suggestion  for  a  pilot  program  was  to  gather  data  from  a  subset  of  PIs  who  already  have  collected  extensive  tracking  information  and  use  that  as  a  guide  for  what  types  of  information  should  be  gathered.  Indeed,  there  was  a  strong  feeling  that  aggregated  data  should  not  supersede  the  personal  stories  collected  by  the  PIs,  and  in  fact,  the  PIs  with  the  personal  connection  to  students  are  more  likely  to  get  helpful  responses  from  former  participants.  Finally,  it  was  emphasized  that  there  are  multiple  successful  outcomes  of  the  REU  program  or  an  REU  site,  and  life  satisfaction  is  a  metric  that  should  be  considered,  not  just  career  placement.    

Communicating  with  REU  Stakeholders  About  Evaluation  Criteria  and  Metrics  for  Success      REU  program  metrics  for  success  

● Essential  to  recognize  the  diversity  of  outcomes  targeted  both  within  any  given  REU  program  and  among  different  REU  programs.  Judging  success,  therefore,  must  be  tailored  to  the  specific  goals  and  emphases  of  the  programs.  

● Positive  experience  in  research  and  satisfaction  with  the  summer  research  experience  ● Students  actively  contribute  to  scholarship  in  the  field  via  publications  and  presentations.  ● Program  fosters  retention  in  STEM.  This  could  include  increasing  the  likelihood  and  numbers  of  students  

transitioning  from  2  to  4  year  colleges,  staying  in  a  STEM  major,  completing  undergraduate  school,  attending  graduate  school,  or  finding  STEM  related  jobs  in  industry,  education,  academia,  government,  etc.  

● Provide  students  with  the  tools  for  success  and  making  informed  decisions  about  their  careers.  ● Positively  influencing  student  self-­‐efficacy  in  doing  science  and  participating  in  a  professional  

community.  ● Determine  whether  the  program  helped  students  focus  on  their  goals  and  understanding  of  a  career  in  

science.  ● Program  has  an  applicant  pool  that  contributes  to  its  specific  goals  for  broadening  participation  (and  not  

just  focus  on  numbers  of  applications).  ● Program  maximizes  the  potential  for  student  growth  by  selecting  students  with  limited  prior  or  current  

opportunities  for  independent  research.  Goals  

● Influence  solicitation  to  guide  future  PI’s  and  review  panels.  Specifically,  we  hope  that  NSF  will  support  a  broad  diversity  of  specific  program  goals  and  approaches,  avoiding  narrow  or  rigid  criteria  for  success.  

○ Action:  Convene  PI  subgroup  with  program  directors  to  provide  input  for  revising  the  program  solicitation  

● Ensure  that  participant  tracking  accurately  reflects  metrics  for  success  for  individual  programs  as  well  as  NSF  goals  

23  

○ Action:  Work  jointly  with  Mathematica  to  ensure  that  metrics  are  appropriately  weighted  for  the  different  programs.  

 One  participant  wrote  that  “the  conversations  about  the  different  ways  in  which  success  could  be  measured  was  extremely  valuable.  It  made  me  think  more  deeply  about  considering  the  purpose  of  individual  programs  in  assessing  success.”    

Strategies  for  Broadening  Participation      Focus  areas:  1.  Continuing  Support  for  REU  Students:  We  are  interested  in  providing  support  for  REU  participants  before,  during,  and  after  their  REU  summer  experience.    Action  Item  1:  Development  of  sustained  collaborations  with  MSIs  and  other  institutions  so  that  their  faculty  can  identify,  mentor,  and  provide  preliminary  research  training  to  potential  applicants,  including  throughout  the  application  process,  as  well  as  supporting  them  after  the  REU  experience.      Action  Item  2:  Creation  of  REU  consortia  (regional  or  topic-­‐based)  or  common  applicant  pools,  to  provide  more  opportunities  to  potential  participants  as  well  as  to  broaden  their  perceptions  of  the  available  opportunities.      Action  Item  3:  Identification  of  ways  that  the  community  around  the  REU  participant  can  be  included  or  supported  (including  immediate  family,  faculty  at  PUIs  or,  in  the  case  of  Native  American  students,  tribal  chiefs).      2.  Getting  the  Word  Out  to  Potential  Participants:  One  key  element  of  broadening  participation  in  REU  programs  is  recruitment.  A  significant  barrier  to  students  applying  is  simply  that  they  don’t  know  about  the  program,  and  this  is  much  more  likely  to  be  an  issue  for  underserved  students.  Raising  awareness  of  the  existence  and  value  of  REUs  also  lends  itself  to  being  addressed  at  the  NSF-­‐wide  level.    Action  Item  1:  Updating  the  NSF  REU  website  to  provide  information  targeted  to  students,  especially  underserved  students.  This  includes  information  on  the  funding  structure  (not  just  stipends,  but  also  travel  support  and  room  and  board),  information  on  the  value  of  REUs  to  post-­‐graduate  careers,  and  sharing  the  experiences  of  REU  participants.    Action  Item  2:  Active  recruitment  at  the  NSF-­‐wide  level:  targeting  PUIs  and  MSIs  with  the  goal  of  creating  wide  awareness  of  the  opportunity  that  an  REU  provides;  recruiting  at  conferences,  particularly  ones  geared  towards  underrepresented  minorities;  development  of  partnerships  with  programs  such  as  LSAMP  and  McNair.    Action  Item  3:  Provision  of  support  to  REU  participants  in  becoming  ambassadors  for  programs  in  their  home  communities  and  institutions.    Action  Item  4:  Encouragement  of  REU  sites  to  engage  in  publicity,  such  as  sending  press  releases  about  students  to  their  home  institutions,  creating  videos  and  other  content  for  social  media,  and  creating  profiles  on  experiences  and  career  paths  of  former  REU  participants.    3.  Changing  the  Framing  Around  Diversity  for  the  REU  Community:  While  the  REU  community  (PIs,  review  panelists,  mentors,  and  program  officers)  can’t  control  what  happens  on  either  side  of  the  programs  (including  

24  

the  pipeline  issue  into  STEM  and  the  lack  of  role  models  in  underrepresented  groups  in  higher  education),  the  REU  community  nevertheless  needs  to  take  responsibility  for  creating  an  inclusive  environment  in  which  all  REU  participants  have  the  opportunity  to  succeed.    Action  Item  1:  Education  of  the  REU  community  about  implicit  bias,  structural  inequities  in  society,  framings  (a  ‘deficit  model’  vs  a  model  that  focuses  on  ‘accumulation  of  advantage’)  and  about  creating  learning  environments  that  are  welcoming  for  all  participants.      Action  Item  2:  Creation  of  selection  metrics  that  reflect  a  diversity  of  backgrounds,  and  including  personal  narratives  (‘distance  travelled’)  as  part  of  the  application  process.    Action  Item  3:  Establishment  of  a  partnership  with  the  National  Research  Mentoring  Network,  which  is  funded  by  the  NIH  to  support  mentor  training,  with  the  goal  of  extending  this  type  of  training  across  all  NSF  directorates.    Action  Item  4:  Leveraging  of  existing  NSF  alliances  for  broadening  participation  in  CS  and  engineering.  

How  Can  the  REU  Group  Inform  the  Agenda  and  Priorities  for  Congress      Action  Item  1:  We  encourage  each  REU  site  to  develop  one  student  highlight  (1  page).  There  should  be  at  least  one  highlight  representing  each  state  available.  One  way  to  begin  drafting  a  highlight  might  be  to  contact  university  or  Council  of  Undergraduate  Research  (CUR)  to  see  if  they  have  template  for  case  study  or  highlights.    Action  Item  2:  We  recommend  that  the  PAN  REU  website  provides  an  online  standard  form  or  template  for  the  student  highlight.      Action  Item  3:  We  would  also  like  to  see  a  list  of  best  practices  for  contacting  congressmen    and  women  with  items  categorized  by  effort  and  linked  with  addresses,  hash  tags,  phone  numbers  for  communicating.    Action  Item  4:  Encourage  the  use  of  Twitter.    Have  a  common  hashtag  for  all  programs  to  use.    We  suggest  #NSFREU.    Action  Item  5:  If  there  is  going  to  be  a  PAN  REU  leadership  group,  they  need  to  establish  a  connection  with  the  Council  of  Undergraduate  Research  (CUR).    Action  Item  6:  Establish  an  Alumni  page  on  PAN  REU  website.        Action  Item  7:  Encourage  alumni  networking  events  at  professional  meetings.    Action  Item  8:  Encourage  REU  students  to  write  letters  to  congress.    Action  Item  9:  Encourage  REU  PIs  to  talk  with  your  congressional  representative’s  staffer  (not  the  actual  congress  person)  and  thank  them  for  supporting  the  money  that  funded  REU’s.    It  complements  them  and  creates  awareness.    Start  at  the  big  picture:  Undergraduate  research  is  important  to  train  students  for  careers  in  science.    Then,  you  may  move  to  the  importance  of  REU  being  one  of  the  vehicles  for  doing  that.        

25  

Action  Item  10:  Integrate  a  good  communication  plan  into  your  REU  curriculum  so  that  students  know  how  to  appropriately  communicate  their  research.    Action  Item  11:  Encourage  the  PAN  LC  to  provide  recommendations  to  Mathematica  about  how  to  fulfill  the  reporting  requirements  for  America  Competes.            

26  

 

 

 

Appendix    

27  

Appendix  1:  Workshop  Agenda    

Time   Event   Location  Thursday,  April  28      

National  Science  Foundation  Atrium:  4201  Wilson  Blvd,  Arlington,  VA  1:00  –  6:00  PM   Registration  and  Name  Badge  Pickup   NSF  Atrium  3:00  –  4:00  PM   Poster  Session  Set  up   NSF  Atrium  4:00  –  6:00  PM   Large  Group:  Poster  Session   NSF  Atrium    

Holiday  Inn  Rosslyn:  1900  Fort  Myer  Drive,  Arlington,  VA  

7:30  –  9:00  PM   Networking  Dinner  at  Holiday  Inn   Vantage  Point  Rooftop  Restaurant  

Friday,  April  29      7:30  –  8:30  AM   Continental  Breakfast  &  Informal  Poster  Session   Rosslyn  Ballroom  8:30  –  8:45  AM   Welcome  &  Meeting  Goals   Rosslyn  Ballroom  8:45  –  10:00  AM   REU  Program  Officer  Panel   Rosslyn  Ballroom  10:00  –  10:15  AM   AM  Break    10:15  –  12:15  PM   Breakout  Groups:  Sharing  Best  Practices   Various  Rooms  12:15  –  1:15  PM   Networking  Lunch  &  Informal  Poster  Session   Rosslyn  Ballroom  1:15  –  2:00  PM   Breakout  Group  Reports     Rosslyn  Ballroom  2:00  –  3:00  PM   Short  Talks:  Collaborative  Efforts  within  Disciplines   Rosslyn  Ballroom  3:00  –  3:15  PM   PM  Break    3:15  –4:00  PM   Presentation:  REU  Common  Assessment  Tool   Rosslyn  Ballroom  4:00  –  5:00  PM   Disciplinary  Discussions:  Common  Assessment  Tool   Various  Rooms  5:00  –  6:00  PM   Informal  Poster  Session     Rosslyn  Ballroom  6:00  PM   Adjourn  for  the  Day   Dinner  on  your  own  Saturday,  April  30      7:30  –  8:30  AM   Continental  Breakfast  &  Informal  Poster  Session   Rosslyn  Ballroom  8:30  –  8:45  AM   Day  3  Welcome    8:45  –  9:45  AM   Action  Planning  Breakout  Sessions   Various  Rooms  9:45  –  10:00  AM   AM  Break     Rosslyn  Ballroom  10:00  –  11:45  AM   Action  Plan  Presentations   Rosslyn  Ballroom  11:45  –  12:00  PM   Wrap  Up   Rosslyn  Ballroom  12:00  PM   Meeting  Adjourned          

28  

Appendix  2:  Workshop  Attendees  Workshop  attendees  were  invited  via  email  by  the  planning  committee  chairpersons.  The  list  of  attendees  is  presented  here.  The  workshop  attendees,  not  including  the  Program  Officers  who  stopped  as  their  schedules  allowed,  are  listed  in  the  table  below.  The  Pan  REU  2016  workshop  was  sponsored  by  a  grant  to  Janet  Branchaw  at  the  University  of  Wisconsin,  Madison  (NSF  DBI-­‐1617287).    

NSF  Dir  

Name   Institution   Email   Role  

BIO   Alan  Berkowitz   Institute  of  Ecosystem  Studies   berkowitza@caryinstitute.org   Attendee  

BIO   Camellia  Okpodu   Norfolk  State  University   cmokpodu@nsu.edu   Attendee  

BIO   Carmen  Domingo   San  Francisco  State  University   cdomingo@sfsu.edu   Attendee  

BIO   Christine  Li   City  College  of  New  York   cli@sci.ccny.cuny.edu   Attendee  

BIO   Dan  Wubah   Washington  and  Lee  University   dwubah@wlu.edu   Organizing  Committee  

BIO   Ian  Billick  Rocky  Mountain  Biological  

Laboratory   director@rmbl.org   Organizing  Committee  

BIO   Janet  Branchaw   University  of  Wisconsin,  Madison   branchaw@wisc.edu   Organizing  Committee  

BIO   Jeremy  Guinn   United  Tribes  Technical  College   jguinn@uttc.edu   Attendee  

BIO   John  Barthell   University  of  Central  Oklahoma   jbarthell@uco.edu   Attendee  

BIO   John  Matsui   University  of  California,  Berkeley   matsui@uclink4.berkeley.edu   Attendee  

BIO   Matthew  Hamilton   Georgetown  University   hamiltm1@georgetown.edu   Attendee  

BIO   Michael  Ceballos   University  of  Minnesota,  Morris   rceballo@morris.umn.edu   Attendee  

BIO   Michelle  Evans-­‐White   University  of  Arkansas   mevanswh@uark.edu   Attendee  

BIO   Michelle  Whaley   University  of  Notre  Dame   michelle.a.whaley.3@nd.edu   Attendee  

BIO   Sue  Carson   North  Carolina  State  University   sue_carson@ncsu.edu   Attendee  

CISE   Anne  Hee  Hiong  Ngu  Texas  State  University,  San  

Marcos   angu@txstate.edu   Attendee  

CISE   Cecelia  (Ziqian)  Dong   New  York  Institute  of  Technology   ziqian.dong@nyit.edu   Attendee  

CISE   Daniela  Raicu   DePaul  University   draicu@cti.depaul.edu   Attendee  

CISE   Dian  Xu   Boise  State  University   dianxiangxu@boisestate.edu   Attendee  

CISE   Fe  Ling  Indiana  University-­‐Purdue  University  Indianapolis   <fengli@iupui.edu>   Attendee  

CISE   Jamie  Payton  University  of  North  Carolina,  

Charlotte   payton@uncc.edu   Attendee  

CISE   Jonathon  Sprinkle   University  of  Arizona   sprinkle@ece.arizona.edu   Attendee  

CISE   Niki  Pissinou   Florida  International  University   pissinou@cis.fiu.edu   Attendee  CISE/ACI   Vetria  Byrd   Purdue  University   vlbyrd@purdue.edu   Attendee  

EHR   Debbie  Chachra   Olin  College  of  Engineering   Debbie.Chachra@olin.edu   Organizing  Committee  

ENG  Allison  Huff  MacPherson   University  of  Arizona   allison@optics.arizona.edu   Organizing  Committee  

ENG   Arash  Esmaili  Zaghi   University  of  Connecticut   zaghi@engr.uconn.edu   Attendee  

29  

ENG   Carolyn  Nichol   Rice  University   cnichol@rice.edu   Attendee  

ENG   Chun  (Chuck)  Zhang   Georgia  Tech   czhang343@isye.gatech.edu   Attendee  

ENG   Jeff  Bokor   University  of  California,  Berkeley   jbokor@eecs.berkeley.edu   Attendee  

GEO   Claire  Raftery   University  of  California,  Berkeley   claire@ssl.berkeley.edu   Attendee  

GEO   Valerie  Sloan   UCAR   vsloan@ucar.edu   Organizing  Committee  GEO/EAR  

Michael  Hubenthal   Incorporated  Research  Institutions  for  Seismology  

Michael.Hubenthal@iris.edu   Attendee  

GEO/OCE   David  Fields  

Bigelow  Institute  of  Ocean  Sciences   dfields@bigelow.org   Attendee  

GEO/OCE   Deana  Erdner   University  of  Texas,  Galveston   derdner@mail.utexas.edu   Attendee  

GEO/OCE   Stephanie  Schroeder   University  of  Southern  California   slschroe@usc.edu   Attendee  

MPS/AST   Charles  Liu   CUNY  College  of  Staten  Island   cliu@mail.csi.cuny.edu   Organizing  Committee  

MPS/AST  

Kathleen  Eastwood   Northern  Arizona  University   kathy.eastwood@nau.edu   Attendee  

MPS/CHE  

Adrian  Roitberg   University  of  Florida   roitberg@UFL.EDU   Attendee  

MPS/CHE  

Elon  Ison   North  Carolina  State  University   eaison@ncsu.edu   Attendee  

MPS/CHE   Holly  Gaede   Texas  A&M  University   hgaede@mail.chem.tamu.edu   Organizing  Committee  

MPS/CHE   Karelle  Aiken   Georgia  Southern  University   kaiken@georgiasouthern.edu   Attendee  

MPS/CHE   Karen  Buchmueller   Furman  University  

karen.buchmueller@furman.edu  

Organizing  Committee  Co-­‐Chair  

MPS/CHE   Linette  Watkins   James  Madison  University   watkinlm@jmu.edu   Attendee  

MPS/CHE  

Paresh  Ray   Jackson  State  University   paresh.c.ray@jsums.edu   Attendee  

MPS/CHE  

Stephen  Jacobson   Indiana  University   jacobson@indiana.edu   Attendee  

MPS/CHE  

Suri  Iyer   Georgia  State  University   siyer@gsu.edu   Attendee  

MPS/DMR   Abhaya  Datye   University  of  New  Mexico   datye@unm.edu   Organizing  Committee  

MPS/DMR   Molly  Kennedy   Clemson  University   mskenne@clemson.edu   Attendee  

MPS/DMR   Sarah  Morgan   University  of  Southern  Mississippi   Sarah.Morgan@usm.edu   Attendee  

MPS/DMS   Darren  Narayan   Rochester  Institute  of  Technology   dansma@rit.edu   Organizing  Committee  

MPS/DMS  

David  Carroll   Florida  Institute  of  Technology   dcarroll@fit.edu   Attendee  

MPS/DMS  

J.  Maurice  Rojas   Texas  A&M  University   rojas@math.tamu.edu   Attendee  

MPS/DMS  

Jan  Rychta   University  of  North  Carolina,  Greensboro  

j_rychta@uncg.edu   Attendee  

30  

MPS/DMS   Kasso  Okoudjou  

University  of  Maryland,  College  Park   kasso@math.umd.edu   Attendee  

MPS/DMS   Lloyd  Douglas   Mathematical  Association  of  

America   ledougl2@uncg.edu   Attendee  

MPS/DMS  

Maria  Isabel  Bueno  Cachadina  

University  of  California,  Santa  Barbara  

mbueno@math.ucsb.edu   Attendee  

MPS/DMS  

Mercedes  Franco   Mathematical  Sciences  Reseach  Institute  

MFranco@qcc.cuny.edu   Attendee  

MPS/PHY  

Boa-­‐An  Li   Texas  A&M  University,  Commerce  

Bao-­‐An.Li@Tamuc.edu   Attendee  

MPS/PHY   Hari  Areti   Old  Dominion  University   areti@jlab.org   Attendee  

MPS/PHY   Myron  Campbell   University  of  Michigan   myron@umich.edu   Attendee  

MPS/PHY   Srinivas  Mann   University  of  Arizona   smanne@physics.arizona.edu   Attendee  

MPS/PHY   Steve  Turley   Brigham  Young  University   turley@byu.edu   Organizing  Committee  

MPS/PHY  

Umesh  Garg   University  of  Notre  Dame   umesh.garg.1@nd.edu   Attendee  

SBE   Rachel  Kallen   University  of  Cincinnati   rachel.kallen@uc.edu   Organizing  Committee  

SBE   Tim  Hawthorne   University  of  Central  Florida   timothy.hawthorne@ucf.edu   Attendee  UW-­‐Mad  

Aaron  Miller   University  of  Wisconsin,  Madison   aaron.miller@wisc.edu   Coordinator  

UW-­‐Mad  

Jessica  Miller   University  of  Wisconsin,  Madison   jdmiller6@wisc.edu   Assessment  Tool  Programmer  

UW-­‐Mad  

Leah  Nell  Adams   University  of  Wisconsin,  Madison   leah.nell.adams@wisc.edu   Assessment  Tool  Project  Manager  

     

31  

Appendix  3:  Workshop  Poster  Presentations    

#   Authors   Institution   Poster  Title  

P01   Alan  R.  Berkowitz  &  Aude  Lochet  

Cary  Institute  of  Ecosystem  Studies  

Assessing  Student  Outcomes  in  the  Translational  Ecology  for  Undergraduates  Program  at  the  Cary  Institute  of  Ecosystem  

Studies  

P02  Janet  Branchaw,  Christine  Pfund  &  Amber  Smith  

University  of  Wisconsin-­‐Madison  

Cultivating  Effective,  Culturally  Responsive  Research  Mentoring  Relationships  

P03   Claire  Raftery   Space  Sciences  Laboratory,  UC  Berkeley  

ASSURE  at  UC  Berkeley's  Space  Science  Lab  -­‐  Providing  an  inclusive  and  supportive  research  environment  through  a  multi-­‐

tiered  mentorship  model  

P04   David  Carroll  &  Munevver  Subasi  

Florida  Institute  of  Technology   BioMath  REU:  Integrating  Biology  and  Mathematics  into  Undergraduate  Research  

P05   David  Fields   Bigelow  Laboratory  for  Ocean  Sciences  

Gulf  of  Maine  and  the  World  Ocean  REU:  Efforts  to  Increase  Minority  Participation  in  

the  Ocean  Sciences  

P06   Daniela  Stan  Raicu   DePaul  University  How  To  Run  A  Successful  REU  Site:  

Perspectives  From  A  Decade  of  Experience  P07   Hari  Areti   Old  Dominion  University   REU  at  Old  Dominion  University  

P08   Holly  C.  Gaede   Texas  A&M  University   Summer  Scholars:  Creating  a  Community-­‐within-­‐a-­‐Community  

P09   Anne  H.H.  Ngu   Texas  State  University   REUIOT:  Multidisciplinary  Research  in  Internet  of  Things  

P10   Michael  Hubenthal   IRIS  Consortium   Under  the  Hood  of  IRIS’s  Distributed  REU  Site  

P11  Valerie  Sloan  &  Rebecca  

Haacker  University  Corporation  for  Atmospheric  Research  

Mentoring  Strategies  for  Strengthening  Self-­‐efficacy  

P12   Jeremy  E.  Guinn   United  Tribes  Technical  College  Impacts  and  Lessons  from  the  Tribal  College  REU  and  RET  Site  Program  

P13   John  Barthell   University  of  Central  Oklahoma  An  International,  Bee  Centered  Research  Experience  for  Undergraduates:  The  First  

Ten  Years  

P14   Karen  Buchmueller  &  Linette  Watkins  

Furman  University  and  James  Madison  University  

Chemistry  REU  Leadership  Group  

P15   Kathy  DeGioia  Eastwood   Northern  Arizona  University  Student-­‐Centered  Cooperation  among  REU  

Sites  in  Astronomy  

P16   Feng  Li  Indiana  University  -­‐Purdue  University  Indianapolis  

Enhancing  Undergraduate  Experience  in  Mobile  Cloud  Securtiy:  SaTC  REU  

P17   Marian  Kennedy   Clemson  University  Tracking  and  Understanding  Student  

Perceptions  of  their  Capability  to  Conduct  Research  

P18   Ruben  Michael  Ceballos   University  of  Minnesota-­‐Morris   REU  Site:  Indigenous  America  to  Indigenous  Borneo  (IAIB)  

P19   J.  Maurice  Rojas   Texas  A&M  University  Algebraic  Biology,  Geometric  Algorithms,  and  Number  Theory  at  the  Texas  A&M  

Mathematics  REU  

P20   Jan  Rychtar  The  University  of  North  Carolina  

-­‐  Greensboro   Math  Biology  REU  at  UNCG  

32  

P21   Sarah  E.  Morgan   University  of  Southern  Mississippi  

Pathways  to  Graduate  School:  University  of  Mississippi  Polymer  Innovation  for  a  

Sustainable  Future  

P22   Srinivas  Manne   University  of  Arizona  

University  of  Arizona  Physics  REU:    Engaging  Community  College  Transfer  Students  in  Experimental  

Research  

P23   Steve  Turley   Brigham  Young  University  Improving  Faculty  Mentoring  in  the  Physics  REU  Program  at  Brigham  Young  University  

P24   Vetria  Byrd   Purdue  University  Undergraduate  Research  and  the  Success  of  the  VisREU  Site:  Promising  Practices  and  

Lessons  Learned  

P25   Arash  Esmaili  Zaghi   University  of  Connecticut  

Research  Opportunity  in  Cyber  and  Civil  Infrastructure  Security  for  Students  with  ADHD:  Developing  Next  Generation  

Creative  Engineers  

P26   Ziqian  (Cecilia)  Dong  New  York  Institute  of  

Technology  

Research  Experience  for  Undergraduates  Site  at  New  York  Institute  of  Technology:  Research  on  Security  of  Mobile  Devices  

and  Wireless  Networks  P27   Karelle  Aiken   Georgia  Southern  University   ALL  IN  for  CEMITURE!  

     

33  

Appendix  4:  Action  Planning  Breakout  Session  Notes    Developing  Collaborations  with  MSI’s    We  see  the  importance  of  creating  synergistic  collaborations  with  rather  than  pipelines  from  minority  serving  institutions.  The  goal  of  these  collaborations  is  to  increase  the  number  of  students  participating  in  STEM  careers  and  the  quality  of  their  experience.  We  hope  to  make  these  students  more  competitive  for  graduate  fellowships  and  STEM  careers.  By  increasing  minority  participation  in  science  we  will  increase  the  pool  of  scientists  and  the  contributions  they  will  make  to  our  disciplines.    We  want  to  foster  productive  research  and  student  training  collaborations.  These  should  be  based  on  authentic  partnerships.    Our  first  task  was  to  define  what  we  meant  by  a  minority-­‐serving  institution.  They  have  some  differences  (even  when  they  serve  the  same  group  of  students),  but  also  some  common  challenges.  We’ll  address  the  common  challenges  here.    Some  of  the  types  of  institutions  we  developed  were  HI  (Hispanic  Institutions),  HSI  (Hispanic    Serving  Institutions),  HBCU  (Historically  Black  Colleges  and  Universities),  and  NSI  (Native  Serving  Institutions).  There  are  formal  definitions  of  these  types  at  the  U.S.  Department  of  Education  Web  Site.  Among  the  NSI’s,  some  are  controlled  by  tribes  (TCU),  some  by  the  federal  government  (BIA),  and  some  are  not  run  by  either  (AANAPISI  and  NASNTI  [Native  American  Pacific  Islander-­‐Serving  Non-­‐Tribal  Institutions]).    We  felt  that  the  best  working  relationships  between  minority  and  majority-­‐serving  institutions  would  be  true  partnerships.  This  gives  the  students  a  chance  to  have  their  mentoring  and  research  relationships  continue  past  the  actual  REU  experience.  Action  Item  1  Create  a  leadership  team  within  the  PanREU  to  collect  and  share  information  information  to  facilitate  these  collaborations.  The  leadership  team  will  be  chosen  from  existing  teams  from  each  NSF  divisions  where  they  exist.  Where  they  don’t  exist,  we  will  ask  for  interested  volunteers  using  mailing  lists  of  REU  site  directors.  Steve  Turley  will  collect  the  list  of  site  director  email  addresses.  Action  Item  2  We  will  create  a  report  of  suggested  practices  for  establishing  effective  practices  for  collaborating  between  MSI  and  majority-­‐serving  institutions.  Camellia  will  create  a  draft  report  by  June  1,  2016.  We  will  then  circulate  a  report  among  our  group  and  produced  a  final  report  by  July  1,  2016  for  publication  on  the  Pan  REU  web  site.  Michael  will  apply  for  a  supplemental  grant  to  provide  support  for  Camellia  to  work  on  this.      Action  Item  3  Organize  a  workshop  on  effective  collaborations  with  minority-­‐serving  institutions  in  conjunction  with  an  existing  national  meeting  such  as  BetaKappaChi,  SACNAS,  or  AISIS.  The  leadership  group  will  organize  the  first  activity  in  2017.  Participants  Steve  Turley,  Brigham  Young  University  Camellia  Okpodu,    

34  

J.  Maurice  Rojas,    Ruben  “Michael”  Ceballos,  University  of  Minnesota  Maribel  Bueno  Cachadina,  UC  Santa  Barbara  Stephen  C.  Jacobson,  Indiana  University  Jan  Rychtar,  UNC  Greensboro  Kasso  Okoudjou,  Univeristy  of  Maryland,  College  Park  Kathy  Eastwood,  Northern  Arizona  University    Developing  Collaborations  with  Community  Colleges    1.  How  do  members  of  the  group  define  success?  What  are  major  commonalities  and  differences  amongst  the  different  divisions,  if  any?  (Should  every  REU  Site  have  such  a  collaboration?)    

● Success  involves  connections  between  community  colleges  and  four  year  colleges  and  universities.  This  relationship  would  provide  research  opportunities  for  both  faculty  and  students  at  community  colleges.  

● With  regards  to  similarities  and  differences  across  disciplines,  all  groups  can  benefit  from  these  connections.  Some  fields  may  be  more  accessible  to  community  college  students  in  their  first  year.    

● While  not  every  site  needs  to  have  a  collaboration  the  entire  REU  community  would  benefit  as  a  whole  from  connections  between  community  colleges  and  four  year  colleges  and  universities.  

 2.  What  are  one  to  three  goals  that  the  group  considers  of  high  importance  pertaining  to  this  topic?      

● Inform  REU  community  about  successful  community  college  or  multiyear  pathway  programs  from  community  colleges  to  graduate  schools?  (Example  -­‐  Partnerships  in  Astronomy  and  Astrophysics  Research  and  Education  (PAARE  -­‐  AST),  Community  College  Undergraduate  Research  Initiative  (CCURI)  )  

● Develop  meaningful  REU  partnerships  with  2-­‐year  schools  (As  Co-­‐PI’s  or  senior  personnel  on  proposals).  Grants  that  provide  support  to  community  colleges  through  a  subaward  will  assist  community  colleges  that  have  limited  grant  office  personnel.  This  include  joint  mentoring  of  students  with  faculty  from  community  colleges  and  four  year  colleges  and  universities.  It  is  important  to  recognize  that  successful  partnerships  need  support  for  community  college  faculty  which  could  be  in  the  form  of  summer  salary  or  travel  support.  Another  idea  is  to  use  other  solicitations  such  as  Improving  Undergraduate  STEM  Education  (IUSE)  and  Advanced  Technological  Education  (ATE)    to  deepen  connections  with  community  colleges.  

 Action  items:    

1. Each  PI  should  establish  connections  with  a  faculty  member  at  a  local  community  college  and  a  faculty  member  at  a  community  college  at  a  distant  campus.  Connections  with  regional  hubs  will  have  a  significant  impact.  

2. Empower  community  college  faculty  to  be  a  Co-­‐PI  or  Senior  personnel  in  the  renewal  and  involve  them  in  the  REU  programming  and  curriculum    

3. Broadening  participation  in  the  newly  established  leadership  group  (REU2N)  and  connect  with  the  Community  College  Undergraduate  Research  Initiative  (CCURI).  

 Participants:    Vetria  Byrd,  Purdue  University  

35  

David  Carroll,  Florida  Institute  of  Technology  Mercedes  Franco,  Mathematical  Science  Research  Institute  Charles  Liu,  CUNY  -­‐  College  of  Staten  Island  Christopher  Meyer,  National  Science  Foundation  Darren  Narayan,  Rochester  Institute  of  Technology  Anne  Ngu,  Texas  State  University  Lina  Patino,  National  Science  Foundation    Chun  (Chuck)  Zhang,  Georgia  Institute  of  Technology    Formalizing  the  Pan  REU  Community  and  Development  of  a  Pan  REU  Website    Note:  These  two  topics  were  combined  due  to  small  numbers  of  people  in  each  group.    Goal  1:  Establish  online  community  and  resource  center    Long-­‐term  goals  include  the  following:  

1. Make  website  and  resources  public  2. Create  a  Listserv  or  Google  Group  that  is  member-­‐based  for  offline  discussion  among  PIs  and  

prospective  PIs    Context:    While  NCAR|UCAR  has  offered  to  host  a  simple  website  for  the  pan-­‐NSF  REU  community,  Val  Sloan  (UCAR)  says  that  they  do  not  have  the  resources  to  coordinate,  create,  and  maintain  website  content  without  additional  support.  Either  another  organization  can  offer  to  host  and  develop  this  website,  or  support  needs  to  be  solicited  from  NSF  to  develop  the    website.  This  project  could  be  considered  as  a  part  of  a  broader  goal  to  create  a  Pan  NSF  REU  Leadership  Team  or  community,  for  example,  through  an  NSF  “Research  Coordination  Network”  award.  Longer-­‐term  support  to  develop  and  maintain  content  on  the  website  could  be  obtained  as  a  part  of  this  larger  effort.    The  first  action  item  should  then  be  to  identify  people  and  institutions  who  would  be  willing  to  write  an  RCN  proposal  to  coordinate  a  Pan-­‐NSF  REU  Network.    Action  Items:    

1. Identify  people  and  institutions  to  write  an  RCN  proposal  to  coordinate  a  pan-­‐NSF  REU  network.    

2. For  the  website,  one  action  item  could  be  to  create  a  task  force  to  lead  on  this  topic,  and  a  chair  or  co-­‐chairs  of  the  task  force.  Names  suggested  as  representatives  from  each  Division  for  the  task  force  were  as  follows:  

a. AST:  Kathy  Eastwood  b. BIO:  Janet  will  identify  c. CHEM:  Karen/Linette  will  identify    d. CISE:  Dianxiang  Xu  suggested  Audrey  Rorrer  e. GEO:  Val  Sloan,  Deanna  Erdner  f. SBE:  Tim  Hawthorne  &  Rachel  Kallen  g. M/PS:  representative  needed    h. ENG:  representative  needed  

The  first  priority  of  this  task  force  would  be  to  work  with  those  writing  an  RCN  proposal  on  what  is  needed  for  defining  the  needs  for  a  pan  NSF  REU  web  page.  

 

36  

3. Contact  NSF  staff  to  identify  who  might  consider  supporting  an  effort  to  develop  a  basic  website  in  the  short-­‐term,  if  that  is  a  priority.  Possible  contacts  at  NSF  include  Program  Director  Corby  Horbis,  the  Program  Officer  REU  working  group,  and  our  Division  Directors.    

4. Identify  an  organization  and  leaders  to  host  and  manage  an  email  listserv.    Topics  that  covered  on  the  website  could  include,  for  example:  

a. Links  to  discipline-­‐based  REU  leadership  group  web  pages  b. Evidenced-­‐based  (best)  practices  c. Mentoring    d. Broadening  Participation  -­‐  recruitment  and  collaboration  e. See  CHEM  topics  -­‐  comprehensive  array  of  tips  and  toolkits  (Karen/Linette)  f. See  CISE  REU  website  tools  and  topics    g. Communicating  with  Congress  and  public  h. Collaborating  with  CUR  

 Goal  2:  PAN  NSF  REU  LEADERSHIP    

a. Establish  Formal  PAN  community  for  all  REUs  across  the  Divisions  b. Long  Term  Advocacy  for  REU  agenda  more  broadly  

 Actions:  

a. Merge/meet/coordinate  with  any  other  pan  task  forces  (REU2N,  Pan  MSI  LG)    Other  notes:  

● Communicating  with  the  broader  REU  community    ● Requesting  resources  from  the  community  ● Timeframe:  Immediately,  and  continuing  _  goal  another  PAN  REU  meeting  in  a  few  years  ● Long-­‐term  sustainability  and  content  maintenance  

 Participants:  Linette  Watkins  Tim  Hawthorne  Deanna  Erdner  Tim  Hawthorne  Dianxiang  Xu  Val  Sloan  Rachel  Kallen    Informing  the  Development  of  Long-­‐Term  Tracking  Tools  Action  Item:  Though  there  were  reservations  about  the  creation  of  an  NSF  ID  for  REU  applicants,  there  was  general  support  for  a  pilot  project  in  a  limited  set  of  REU  sites,  perhaps  within  a  directorate  or  discipline.  In  general,  the  thought  is  the  student  would  interface  with  an  NSF  portal  to  enter  demographic  information  and  receive  an  ID,  perhaps  in  the  form  of  an  email  address.  This  ID  would  be  used  when  applying  to  individual  programs,  students  would  enter  this  ID  and  demographic  data  would  automatically  upload  to  applications,  saving  the  students  the  task  of  entering  the  same  demographic  data  in  multiple  applications.  Individual  sites  would  report  the  IDs  of  all  applicants  to  the  NSF,  who  would  ultimately  return  data  to  the  site  in  order  for  the  evaluate  aspects  of  their  sites,  for  example  the  success  of  their  recruiting  methods.  The  NSF  ID  could  be  

37  

presumably  used  when  participants  later  interfaced  with  the  NSF  (e.g.  through  applications  to  NSF  GRFP,  CAREER  grants,  and  other  later  individual  grants.)  Needed  Resources:    NSF  has  the  resources  at  their  end  to  create  the  portal  through  existing  contracts  and  the  Office  of  Assessment.  However,  PIs  would  encounter  additional  costs  for  programming  for  modification  of  their  applications.  Workshops  and  meetings  between  contractors  and  the  pilot  group  would  need  to  be  supported  for  implementation,  development,  improvement,  and  assessment  of  the  project.    Timescale:    Since  most  applications  are  opened  in  the  Fall,  it  is  probably  too  late  to  implement  this  for  the  next  application  cycle,  though  workshops  could  be  convened  for  further  discussion  such  that  the  tools  would  be  in  place  for  applications  for  programs  run  during  the  Summer  of  2018.    Additional  notes:  One  concern  about  this  plan  including  hesitation  at  broadening  the  scope  of  the  project  outside  what  the  America  Competes  Act  calls  for,  i.e.  tracking  applicants,  not  just  participants.  Also,  the  REU  community  are  hesitant  about  being  diverted  to  the  task  of  evaluating  the  impact  of  other  NSF  programs.  There  were  also  some  concerns  about  privacy.  Additionally,  some  people  thought  that  some  of  this  data  could  be  obtained  in  other  ways,  for  example,  by  asking  students  to  how  many  programs  they  applied  or  by  mining  the  existing  data  that  are  collected  by  PIs.  In  fact,  an  additional  suggestion  for  a  pilot  program  was  to  gather  data  from  a  subset  of  PIs  who  already  have  collected  extensive  tracking  information  and  use  that  as  a  guide  for  what  types  of  information  should  be  gathered.  Indeed,  there  was  a  strong  feeling  that  aggregated  data  should  not  supercede  the  personal  stories  collected  by  the  PIs,  and  in  fact,  the  PIs  with  the  personal  connection  to  students  are  more  likely  to  get  helpful  responses  from  former  participants.  Finally,  it  was  emphasized  that  there  are  multiple  successful  outcomes  of  the  REU  program  or  an  REU  site,  and  life  satisfaction  is  a  metric  that  should  be  considered,  not  just  career  placement.      Participants:  Janet  Branchaw  (BIO  University  of  Wisconsin)  Holly  Gaede  (CHEM  Texas  A&M  University)  Ziqian  (Cecilia)  Dong  (CISE  New  York  Institute  of  Technology)    Jeremy  Guinn  (BIO  United  Tribes  Technical  College)  David  Fields  (GEO/OCE  Bigelow  Institute  of  Ocean  Sciences)  Corby  Horvis  (NSF)  Michael  Hubenthal  (GEO/EAR  Incorporated  Research  Institutions  for  Seismology)  Jessica  Miller  (University  of  Wisconsin)  Michelle  Whaley  (BIO  University  of  Notre  Dame)  Stephanie  Schroeder  (GEO/OCE  University  of  Southern  California)  Mandy  Simcox  (NSF)      Communicating  with  REU  Stakeholders  About  Evaluation  Criteria  and  Metrics  for  Success  REU  program  metrics  for  success  

● Essential  to  recognize  the  diversity  of  outcomes  targeted  both  within  any  given  REU  program  and  among  different  REU  programs.  Judging  success,  therefore,  must  be  tailored  to  the  specific  goals  and  emphases  of  the  programs.  

● Positive  experience  in  research  and  satisfaction  with  the  summer  research  experience  ● Students  actively  contribute  to  scholarship  in  the  field  via  publications  and  presentations.  ● Program  fosters  retention  in  STEM.  This  could  include  increasing  the  likelihood  and  numbers  of  students  

transitioning  from  2  to  4  year  colleges,  staying  in  a  STEM  major,  completing  undergraduate  school,  

38  

attending  graduate  school,  or  finding  STEM  related  jobs  in  industry,  education,  academia,  government,  etc.  

● Provide  students  with  the  tools  for  success  and  making  informed  decisions  about  their  careers.  ● Positively  influencing  student  self-­‐efficacy  in  doing  science  and  participating  in  a  professional  

community.  ● Determine  whether  the  program  helped  students  focus  on  their  goals  and  understanding  of  a  career  in  

science.  ● Program  has  an  applicant  pool  that  contributes  to  its  specific  goals  for  broadening  participation  (and  not  

just  focus  on  numbers  of  applications).  ● Program  maximizes  the  potential  for  student  growth  by  selecting  students  with  limited  prior  or  current  

opportunities  for  independent  research.  Goals  

● Influence  solicitation  to  guide  future  PI’s  and  review  panels.  Specifically,  we  hope  that  NSF  will  support  a  broad  diversity  of  specific  program  goals  and  approaches,  avoiding  narrow  or  rigid  criteria  for  success.  

○ Action:  Convene  PI  subgroup  with  program  directors  to  provide  input  for  revising  the  program  solicitation  

● Ensure  that  participant  tracking  accurately  reflects  metrics  for  success  for  individual  programs  as  well  as  NSF  goals  

○ Action:  Work  jointly  with  Mathematica  to  ensure  that  metrics  are  appropriately  weighted  for  the  different  programs.  

Participants:  Abhaya  K.  Datye  University  of  New  Mexico  Alan  R.  Berkowitz,  Cary  Institute  of  Ecosystem  Studies  Karelle  Aiken,  Georgia  Southern  University  Chris  Li,  City  College  of  New  York  –  City  University  of  New  York  Allison  Huff  Mac  Pherson,  University  of  Arizona    One  participant  wrote  that  “the  conversations  about  the  different  ways  in  which  success  could  be  measured  was  extremely  valuable.  It  made  me  think  more  deeply  about  considering  the  purpose  of  individual  programs  in  assessing  success.”      Strategies  for  Broadening  Participation    One  of  the  mandates  of  the  REU  program  is  to  increase  the  number  of  Americans  who  are  engaged  in  STEM  (including  industry  and  research).  A  significant  element  of  accomplishing  this  is  broadening  participation  in  STEM  and  related  fields;  in  particular,  because  research  experience  is  increasingly  required  for  admission  into  graduate  school,  REUs  are  de  facto  gatekeepers  and  therefore  have  an  increased  obligation  not  to  reify  existing  inequities.  This  breakout  group  focused  on  ways  the  NSF-­‐wide  REU  community  to  engage  with  and  address  structural  issues  around  representation  and  diversity.    Three  primary  focus  areas  were  identified:  1.  Continuing  Support  for  REU  Students    We  are  interested  in  providing  support  for  REU  participants  before,  during,  and  after  their  REU  summer  experience.  Specific  action  items  include:    

39  

Development  of    sustained  collaborations  with  MSIs  and  other  institutions  so  that  their  faculty  can  identify,  mentor,  and  provide  preliminary  research  training  to  potential  applicants,  including  throughout  the  application  process,  as  well  as  supporting  them  after  the  REU  experience.      Creation  of  REU  consortia  (regional  or  topic-­‐based)  or  common  applicant  pools,  to  provide  more  opportunities  to  potential  participants  as  well  as  to  broaden  their  perceptions  of  the  available  opportunities.      Identification  of  ways  that  the  community  around  the  REU  participant  can  be  included  or  supported  (including  immediate  family,  faculty  at  PUIs  or,  in  the  case  of  Native  American  students,  tribal  chiefs).      2.  Getting  the  Word  Out  to  Potential  Participants    One  key  element  of  broadening  participation  in  REU  programs  is  recruitment.  A  significant  barrier  to  students  applying  is  simply  that  they  don’t  know  about  the  program,  and  this  is  much  more  likely  to  be  an  issue  for  underserved  students.  Raising  awareness  of  the  existence  and  value  of  REUs  also  lends  itself  to  being  addressed  at  the  NSF-­‐wide  level.  Specific  action  items  include:    Updating  the  NSF  REU  website  to  provide  information  targeted  to  students,  especially  underserved  students.  This  includes  information  on  the  funding  structure  (not  just  stipends,  but  also  travel  support  and  room  and  board),  information  on  the  value  of  REUs  to  post-­‐graduate  careers,  and  sharing  the  experiences  of  REU  participants.    Active  recruitment  at  the  NSF-­‐wide  level:  targeting  PUIs  and  MSIs  with  the  goal  of  creating  wide  awareness  of  the  opportunity  that  an  REU  provides;  recruiting  at  conferences,  particularly  ones  geared  towards  underrepresented  minorities;  development  of  partnerships  with  programs  such  as  LSAMP  and  McNair.    Provision  of  support  to  REU  participants  in  becoming  ambassadors  for  programs  in  their  home  communities  and  institutions.    Encouragement  of  REU  sites  to  engage  in  publicity,  such  as  sending  press  releases  about  students  to  their  home  institutions,  creating  videos  and  other  content  for  social  media,  and  creating  profiles  on  experiences  and  career  paths  of  former  REU  participants.      3.  Changing  the  Framing  Around  Diversity  for  the  REU  Community      While  the  REU  community  (PIs,  review  panelists,  mentors,  and  program  officers)  can’t  control  what  happens  on  either  side  of  the  programs  (including  the  pipeline  issue  into  STEM  and  the  lack  of  role  models  in  underrepresented  groups  in  higher  education),  the  REU  community  nevertheless  needs  to  take  responsibility  for  creating  an  inclusive  environment  in  which  all  REU  participants  have  the  opportunity  to  succeed.  Specific  action  items  include:    Education  of  the  REU  community  about  implicit  bias,  structural  inequities  in  society,  framings  (a  ‘deficit  model’  vs  a  model  that  focuses  on  ‘accumulation  of  advantage’)  and  about  creating  learning  environments  that  are  welcoming  for  all  participants.    

40  

 Creation  of  selection  metrics  that  reflect  a  diversity  of  backgrounds,  and  including  personal  narratives  (‘distance  travelled’)  as  part  of  the  application  process.    Establishment  of  a  partnership  with  the  National  Research  Mentoring  Network,  which  is  funded  by  the  NIH  to  support  mentor  training,  with  the  goal  of  extending  this  type  of  training  across  all  NSF  directorates.    Leveraging  of  existing  NSF  alliances  for  broadening  participation  in  CS  and  engineering.    How  Can  the  REU  Group  Inform  the  Agenda  and  Priorities  for  Congress    

● We  encourage  each  REU  site  to  develop  one  student  highlight  (1  page).    There  should  be  at  least  one  highlight  representing  each  state  available.  One  way  to  begin  drafting  a  highlight  might  be  to  contact  university  or  Council  of  Undergraduate  Research  (CUR)  to  see  if  they  have  template  for  case  study  or  highlights.  

● We  recommend  that  the  PAN  REU  website  provides  an  online  standard  form  or  template  for  the  student  highlight.      

● We  would  also  like  to  see  a  list  of  best  practices  for  contacting  congressmen    and  women  with  items  categorized  by  effort  and  linked  with  addresses,  hash  tags,  phone  numbers  for  communicating.  

● Encourage  the  use  of  Twitter.    Have  a  common  hashtag  for  all  programs  to  use.    We  suggest  #NSFREU.  ● If  there  is  going  to  be  a  PAN  REU  leadership  group,  they  need  to  establish  a  connection  with  the  

Council  of  Undergraduate  Research  (CUR).  ● Establish  an  Alumni  page  on  PAN  REU  website.      ● Encourage  alumni  networking  events  at  professional  meetings.  ● Encourage  REU  students  to  write  letters  to  congress.  ● Encourage  REU  PIs  to  talk  with  your  congressional  representative’s  staffer  (not  the  actual  congress  

person)  and  thank  them  for  supporting  the  money  that  funded  REU’s.    It  complements  them  and  creates  awareness.    Start  at  the  big  picture:  Undergraduate  research  is  important  to  train  students  for  careers  in  science.    Then,  you  may  move  to  the  importance  of  REU  being  one  of  the  vehicles  for  doing  that.      

● Integrate  a  good  communication  plan  into  your  REU  curriculum  so  that  students  know  how  to  appropriately  communicate  their  research.  

● Encourage  the  PAN  LC  to  provide  recommendations  to  Mathematica  about  how  to  fulfill  the  reporting  requirements  for  America  Competes.  

   Participants:  Ian  Billick  Sally  O’Connor  Michelle  Evans-­‐White  Susan  Carson  Carmen  Domingo  Carolyn  Nichol,  Adrian  Roitberg  Arash  Esmaili  Zaghi  Lloyd  Douglas  Daniela  Raicu      

41  

Appendix  5:  Workshop  Evaluation  Results    For  the  pre-­‐workshop  survey,  43  people  responded.      For  the  post-­‐workshop  survey,  61  people  responded.    The  results  of  the  pre-­‐survey  and  post-­‐survey  are  included  below.    

2016  Pan-­‐NSF  REU  Workshop  Pre-­‐survey    1.  

By  clicking  the  “yes”  radio  button  below,  you  give  your  consent  to  participate  in  this  project.  

Answer  Options  Response  Percent   Response  Count  

Yes   100.0%   43  No   0.0%   0    2.  

How  many  years  has  your  program  been  in  existence,  in  total?  

Answer  Options   Response  Count  

    44  Response  Text  2,  29,  3.5,  15,  8,  6,  10,  12,  4,  8,  25,  18,  6,  20,  5,  3,  14,  1,  9,  1,  7,  17,  1.5,  25,  4,  13,  24,  12,  30,  3,  7,  20,  13,  4,  22,  5,  8,  5,  13,  2,  25,  7,  5,  30    3.  

Which  NSF  Division  funds  your  REU?  (May  select  more  than  one,  if  applicable)  

Answer  Options   Response  Percent  

Response  Count  

BIO   22.7%   10  CISE   11.4%   5  CISE/ACI   4.5%   2  EHR   4.5%   2  ENG   9.1%   4  GEO   0.0%   0  GEO/AGS   2.3%   1  GEO/EAR   4.5%   2  GEO/OCE   9.1%   4  MPS/AST   4.5%   2  MPS/CHE   11.4%   5  MPS/DMR   0.0%   0  MPS/DMS   13.6%   6  MPS/PHY   9.1%   4  SBE   0.0%   0  Other  (please  specify)   4  

42  

 

   4.    Which  of  the  following  do  you  include  as  formal  components  of  your  REU  program  (e.g.  in  a  workshop,  seminar,  activity)?  (Select  all  that  apply)  

Answer  Options   Response  Percent  

Response  Count  

Research  ethics  training   81.8%   36  Information  literacy   45.5%   20  Graduate  school  information  or  preparation   81.8%   36  Laboratory  safety  training   54.5%   24  Scientific  communication   93.2%   41  Proposal  writing   29.5%   13  Education  and  outreach   50.0%   22  Cohort-­‐  or  team-­‐building   59.1%   26  Other  (please  specify):   13    

Other  (please  specify):  

Experimental  Statistics  careers  in  marine  science  Native  American  specific  workshops/guest  speakers  Computer  programming  for  scientists,  abstract  writing,  scientific  poster  design  and  construction,  Networking,  etc.    weekly  research  seminar  I  run  the  BIO-­‐OCE  travel  scholarship  program,  not  a  PI  on  an  REU  currently  International  research  experience  Workshops  on  creativity,  reinforcement  of  learning  differences.  Brainstorming  sessions.  field  trips,  museum  tours  Field  trips,  Physics  Olympics,  GRE  Prep,  Computer  Course  Heuristics  in  Picking  up  Specific  Research  Skills,  How  to  apply  and  Succeed  in  Graduate  School  Technical  &  Professional  Seminars,  Invited  Talks,  Emerging  Research,  Journal  Club  Mathematical  research  5-­‐day  training  in  model  and  non-­‐model  species  and  a  3-­‐day  camping  trip  to  learn  about  local  habitats  and  ecological  issues.  

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

Which NSF Division funds your REU? (May select more than one, if applicable)

43  

   

   5.  

Please  indicate  whether  the  following  statement  is  true,  to  the  best  of  your  knowledge.  

Answer  Options   Yes   No  Unsure/don't  

know  

Response  

Count  I  attended  the  pan-­‐NSF  REU  workshop  in  2005   6   36   3   45  My  discipline  organizes  oral  or  poster  sessions  on  REU  practices  at  national  science  meetings   19   6   19   44  

My  discipline  has  an  REU  leadership  group  or  committee   23   6   15   44  I  have  attended  an  REU  PI  meeting  in  my  discipline   32   13   0   45  Our  REU  site  is  multidisciplinary  (e.g.  across  NSF  Divisions  or  Directorates)  

8   35   1   44  

Other  (please  specify)   2    

Other  (please  specify)  

I  attended  a  Pan-­‐REU  PI  meeting  several  years  back,  perhaps  the  most  recent  one?  Our  REU  site  is  under  the  Chemistry  Division  but  we  also  have  chemistry-­‐related  projects  in  Biology  and  Public  Health.      

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0%

100.0%

Res

earc

h et

hics

trai

ning

Info

rmat

ion

liter

acy

Gra

duat

e sc

hool

in

form

atio

n or

pr

epar

atio

n

Labo

rato

ry

safe

ty tr

aini

ng

Sci

entif

ic

com

mun

icat

ion

Pro

posa

l w

ritin

g

Edu

catio

n an

d ou

trea

ch

Coh

ort-

or

team

-bui

ldin

g

Which of the following do you include as formal components of your REU program (e.g. in a workshop, seminar, activity)? (Select all that apply)

44  

   6.  

Please  indicate  how  valuable  you  would  consider  each  of  these  potential  Pan-­‐REU  workshop  topics.  

Answer  Options  Not  

valuable   Valuable  Very  

valuable   Unsure  Rating  Average  

Response  Count  

Developing  an  REU  PI  community  within  each  discipline  

4   16   19   4   2.53   43  

Sharing  REU  strategies  and  tools  across  disciplines   1   11   30   1   2.72   43  

Development  of  REU  leadership  groups   6   13   17   9   2.64   45  Common  application  development  and  implementation  

4   19   16   6   2.53   45  

Strategies  for  broadening  participation   1   9   35   0   2.76   45  Mentor  training  and  preparation   2   13   29   1   2.64   45  Ethics  and  the  responsible  conduct  of  research  training   3   24   14   4   2.42   45  

Student  cohort-­‐  or  team-­‐buliding   5   21   13   6   2.44   45  Creating  a  safe  and  inclusive  environment   3   20   21   1   2.44   45  Career  exploration  and  preparation  for  students  

1   20   20   3   2.57   44  

Evaluation  and  assessment  of  REU  sites   0   11   33   1   2.78   45  Impact  of  REU  programs  and  tracking  REU  participants   0   7   38   0   2.84   45  

International  REUs   11   13   9   11   2.45   44  Other  (please  specify)   3    

Other  (please  specify)  

New  look  at  "disability",  by  emphasizing  on  strengths  and  differences.    At  some  places  faculty  may  handle  responsibilities  that  are  typically  delegated  to  full-­‐time  staff  at  larger  schools.  How  do  PI’s  at  PUI’s  for  example,  effectively  manage  the  administrative  responsibilities  of  the  REU?  What  are  some  of  their  best  

0 5

10 15 20 25 30 35 40 45 50

I atte

nded

the

pan-

NS

F R

EU

w

orks

hop

in 2

005

My

disc

iplin

e ha

s an

RE

U le

ader

ship

gr

oup

or

com

mitt

ee

Our

RE

U s

ite is

m

ultid

isci

plin

ary

(e.g

. acr

oss

NS

F

Div

isio

ns o

r D

irect

orat

es)

Please indicate whether the following statement is true, to the best of your knowledge.

Yes No Unsure/don't know

45  

practices?  Given  that  this  is  a  PAN  meeting,  it  seems  like  it  should  focus  on  items  which  are  not  redundant  with  the  PI  meeting,  and  which  are  truly  Pan  REU    

   7.  Please  describe  areas  or  topics  that  you  believe  that  your  REU  (or  REUs  in  your  discipline)  are  especially  strong  in.  Interdisciplinary  research,  technical  writing,  professional  development,  public  speaking  (conferences  and  poster  presentations),    Giving  an  authentic  research  experience  in  mathematics,  giving  students  experience  in  writing  research  papers  and  giving  presentations,  graduate  school  preparation.  Student  reflection/metacognition  Scientific  communication  Human  subjects  research  Research  preparation  Broadening  participation  (URM  group).  Student  Cohort  and  team  building.  

Supporting  students  as  hey  apply  to  and  go  through  graduate  school.  Research  mentoring,  creating  a  safe  inclusive  space,  RCR.  Our  REU  targets  community  college  students  and  we  have  been  effective  in  engaging  and  retaining  these  students  in  STEM  Accelerator  Physics  and  Technology  and  Experimental  and  Theoretical  Nuclear  Physics    exposing  students  to  graduate  school  environment  career  exploration  Virtual  co-­‐horts  and  collaboration,  evaluation,  mentoring  process  Bio  has  a  wide  variety  of  programs,  so  there  are  strengths  across  many  dimensions.  We  use  a  common  assessment  tool.  As  a  former  member  and  chair  of  the  BIO  REU  leadership  committee  and  remaining  in  touch  since  I  left  the  council  I  am  particularly  pleased  with  the  BIO  REU  leadership  efforts  which  are  way  ahead  of  others  in  PI  and  student  support  and  coordination.  Improvements  in  self-­‐efficacy  for  research  

2.20 2.40 2.60 2.80 3.00

Developing an REU PI community within each

Development of REU leadership groups

Strategies for broadening participation

Ethics and the responsible conduct of research

Creating a safe and inclusive environment

Evaluation and assessment of REU sites

International REUs

Please indicate how valuable you would consider each of these potential Pan-REU workshop topics.

46  

Student  development  Scientific  Communication  Team  building  Focusing  on  unique  potentials  of  a  group  of  students    (with  ADHD)  who  are  traditionally  undeserved  in  engineering  programs.  Strong  cohort  building.  Recruitment  cohort  building  broadening  participation    safe  and  inclusive  environment  leadership  development  Recruitment.    The  NSF  Chemistry  Division  has  done  a  good  job  of  increasing  the  number  of  sites  across  the  country  and  at  different  types  of  institutions.  The  PI's  meeting  last  summer  for  Chemistry  was  one  of  the  best  workshops  that  I  have  ever  attended  in  my  career.  As  a  new  PI  at  the  time,  I  found  the  sessions  extremely  useful  and  I  have  modified  quite  a  few  things  in  my  REU  based  on  information  I  received  from  the  workshop.  It  was  very  practical.  The  leadership  committee  in  Chemistry  is  visible  and  reaches  out  PI’s  on  a  regular  basis.  I  appreciated  that  they  offered  to  advertise  all  the  REU’s  at  SACNAS.  

Publishable  research  projects.  Excellent  mentors.  Many  female  mentors.    Our  REU  is  very  strong  at  providing  experience  in  translational  research  and  interaction  with  industry  and  government  groups.  collaborating  with  other  undergraduate  research  programs  based  at  the  same  institution  to  create  a  larger,  broader  student  research  community.  medical  informatics  Network  and  mobile  device  security  building  team/  cohort,  assessment,  ethics  training  Exposing  students  from  4  year  institutions  to  research,  encouraging  them  to  follow  a  path  to  a  career  in  science  Training  camp  (including  machine  shop,  digital  electronics  etc.)  and  close  mentoring  and  intervention  (as  necessary)  during  research  phase  Responsible  conduct  of  research;  working  with  Native  American  students  Allowing  students  to  actually  participate  in  research  that  is  publishable.  Giving  research  topics  to  students  leading  to  publishable  results.  Our  programs  involve  a  number  of  very  effective  mentors  working  on  a  variety  of  subfields    in  mathematics.  Therefore,  we  are  able  to  offer  students  a  number  of  projects  with  various  level  of  difficulties  and  preparation.    REU  student  composition  -­‐  ethnic  diversity  as  well  as  diversity  with  respect  to  the  institutions  the  students  come  from  (community  college,  less  resource  universities  and  colleges).  Our  research  area  is  the  intersection  of  ecology,  evolution  and  development  -­‐  our  strength  is  in  how  changes  in  climate  may  impact  animal  and  plant  populations  especially  in  estuary  and  marine  habitats.  Linear  Algebra,  Combinatorics.    Career  planning,  experimental  design  and  implementation    8.    Approximately  what  percentage  of  your  REU  participants  have  been  from  a  group  that  is  historically  underrepresented  in  STEM?  

Answer  Options   Response  Percent  

Response  Count  

0  -­‐  9%   0.0%   0  10  -­‐  24%   8.9%   4  25  -­‐  49%   42.2%   19  

47  

50  -­‐  74%   26.7%   12  75  -­‐  100%   17.8%   8  Other  (please  specify):   4.4%   2    

   9.  

What  program  assessment  tool(s)  do  you  currently  use  for  program  evaluation?  (Select  all  that  apply)  

Answer  Options  Response  Percent   Response  Count  

SURE  III   9.3%   4  SALG/URSSA   30.2%   13  Our  own  surveys   76.7%   33  Other  (please  specify)   8    

Other  (please  specify)  

external  evaluator  uses  tools  including  surveys,  focus  groups  External  evaluator  Bio  REU  Common  Assessment  Tool  (from  URSSA)  formative  evaluations,  exit  interviews  rubric  on  critical  and  creative  thinking    CISE  has  assessment  tools  that  we  can  modify  to  fit  our  programs  Exit  interviews  with  associate  dept  heads  We  also  have  a  UMD's  education  faculty  evaluate  our  program.  

Approximately what percentage of your REU participants have been been from a group that i historically underrepresented in STEM?

0 - 9%

10 - 24%

25 - 49%

50 - 74%

75 - 100%

48  

   10.  

What  tools  do  you  use  to  track  or  provide  support  to  REU  participants  over  time?  Check  all  that  apply.  

Answer  Options   Response  Percent  

Response  Count  

Social  media  (like  FaceBook,  etc.)   63.6%   28  Direct  contact  via  email,  text,  or  phone   100.0%   44  Voluntary  reporting  by  students  through  on-­‐line  tools   38.6%   17  We  do  not  track  participants  after  they  leave  the  program.   0.0%   0  Other  (please  specify)   2    

Other  (please  specify)  

tracking  database  and  web  portal  built  by  my  institution  We  also  include  our  REU  students  in  emails  that  we  send  out  to  our  own  chemistry  majors  regarding  internships  and  job  opportunities.    

   11.    

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

SURE III SALG/URSSA Our own surveys

What program assessment tool(s) do you currently use for program evaluation? (Select all that apply)

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

120.0%

Social media (like FaceBook, etc.)

Direct contact via email, text, or phone

Voluntary reporting by students through

on-line tools

We do not track participants after they

leave the program.

What tools do you use to track or provide support to REU participants over time? Check all that apply.

49  

Please  indicate  how  effective  you  think  each  of  these  mechanisms  would  be  for  REU  PIs  to  share  resources  and  strategies  across  disciplines.  

Answer  Options   Not  valuable  

Valuable   Very  valuable   Unsure   Rating  Average  

Response  Count  

An  email  listserv  for  REU  PIs  across  disciplines   7   22   11   5   3.51   45  

Telecons  on  relevant  topics   6   20   6   12   3.82   44  Sessions  (oral,  poster)  at  national  science  meetings  

6   19   13   7   3.78   45  

Workshops  at  national  science  meetings   2   26   15   2   3.71   45  

Resources  on  the  web  provided  by  PIs   0   19   23   3   4.22   45  

Other  (please  specify)   1    

Other  (please  specify)  

a  wiki  or  database    

   12.  Do  you  have  any  comments  or  suggestions  regarding  this  pan-­‐NSF  REU  PI  workshop?  

Response  Text  

I'm  very  much  looking  forward  to  it.  Developing  and  sharing  strategies  to  reach  out  to  economically  disadvantaged  students  would  be  valuable  along  with  developing  tools  for  effective  mentoring  of  the  students,    -­‐  A  discussion  of  sharing  evaluation  data  or  making  white  papers  available  online.    -­‐  Publications  are  currency  for  many...  therefore  a  discussion  of  how/where  to  publish  about  your  REU  would  be  useful.    -­‐  Developing  a  definition  of  what  the  "REU"  model  currently  is  (this  has  evolved)  and  what  are  the  current  spectrum  of  variants  (mentoring,  student  interactions,  etc).  This  could  be  followed  with  providing  opportunities  for  various  groups  

0.00 1.00 2.00 3.00 4.00 5.00

An email listserv for REU PIs across

Telecons on relevant topics

Sessions (oral, poster) at national science

Workshops at national science meetings

Resources on the web provided by PIs

Please indicate how effective you think each of these mechanisms would be for REU PIs to share resources and strategies across disciplines.

50  

outside  the  model  to  interact.  It  could  also  lead  to  a  discussion  of  testing  what  aspects  of  the  model  have  the  most  impact  for  what  sorts  of  audiences.  I  raise  this  as  a  long  running  REU  site  that  must  continuously  evolve  and  test  new  ideas  to  maintain  funding.  In  the  past  this  has  primarily  focused  on  evolving  our  own  site,  However,  we  are  now  interested  in  working  with  others  to  look  beyond  our  own  REU  and  test  the  model/theory  of  the  REU  itself.  Of  course  a  related  discussion  would  also  be  funding  mechanisms  to  support  more  robust  "testing"  than  is  currently  available  through  REU  Site  awards  which  are  designed  to  primarily  support  students.    No  Thank  you  for  your  efforts.  We,  as  a  community,  should  use  this  workshop  to  communicate  with  NSF  and  our  POs  to  express  what's  great,  what's  working,  and  what  needs  improvement.  Include  icebreaker  activities  that  can  help  PIs  to  get  to  know  each  other  quicker.  This  is  my  first  one,  so  I  don't  have  many  suggestions.    I  have  been  very  proud  of  the  BIO-­‐REU's  community  and  leadership.    I  always  walk  away  being  ahead  of  the  game.    I  am  hoping  that  is  the  same  with  this  workshop.  While  I  like  the  idea  of  creating  tools  for  REU  programs  (e.g.,  managing  applications,  assessment),  I  think  requiring  the  use  of  those  tools  can  create  a  lot  of  problems  and  leads  to  enforced  mediocrity,  unless  there  is  a  clearly  defined  and  accepted  endpoint  for  requiring  the  use  (e.g.,  specific  data  to  help  bolster  the  case  for  continuing  the  program).  Not  at  this  time    13.  

Which  race/ethnicity  best  describes  you.    Check  all  identities  that  apply.  

Answer  Options   Response  Percent  

Response  Count  

American  Indian,  Alaskan  Native,  Native  Hawaiian,  or  Pacific  Islander   4.5%   2  

Black  or  African  American   13.6%   6  Asian  (including  South  Asian)   22.7%   10  Latino  or  Hispanic   13.6%   6  Caucasion   45.5%   20  Other  (please  specify)   1    

   14.    Please  indicate  your  gender:  

Which race/ethnicity best describes you. Check all identities that apply.

American Indian, Alaskan Native, Native Hawaiian, or Pacific Islander Black or African American

Asian (including South Asian)

Latino or Hispanic

Caucasion

51  

Answer  Options   Response  Percent   Response  Count  Female   47.7%   21  Male   52.3%   23  Alternative  (please  specify):   0    

   

Please indicate your gender:

Female Male

 

52  

2061  Pan-­‐NSF  REU  Workshop  Post-­‐survey    1.    By  clicking  the  “yes”  radio  button  below,  you  give  your  consent  to  participate  in  this  project.  

Answer  Options   Response  Percent   Response  Count  

Yes   98.4%   61  No   1.6%   1    2.  Overall,  how  would  you  rate  the  2016  Pan-­‐NSF  REU  Workshop?  

Answer  Options   Response  Percent   Response  Count  

Not  valuable   1.6%   1  Valuable   22.6%   14  Very  valuable   75.8%   47  Unsure   0.0%   0    

   3.    

What  was  most  valuable  about  this  workshop  for  you?  

Answer  Options   Response  Count  

    57    Response  Text  

Sharing  of  best  practices  chance  to  interact  with  experienced  REU  Site  directors  from  so  many  different  disciplines  Hearing  other  ideas  to  use  in  my  own  project.      Hearing  from  NSF  Program  Officers  what  they  are  looking  to  fund  and  talking  with  other  REU  PI's  

Overall, how would you rate the 2016 Pan-NSF REU Workshop?

Not valuable Valuable

 

53  

networking    Meeting  REU  PI's  across  disciplines.  Learning  about  their  challenges  and  successes.  The  discussions  on  how  to  increase  diversity  among  one's  REU  participants.    Sharing  ideas  with  many  wise  program  leaders  and  NSF  program  officers.  Concrete  planning  for  initiatives.  Stretching  my  understanding  and  ideas  about  undergraduate  research  by  learning  about  how  it  unfolds  in  other  disciplines  and  settings.  participant  tracking  and  evalaution  The  brainstorming  sessions  with  other  PIs  of  REU  sites  is  invaluable  to  continued  positive  development  of  my  site  over  time.  I  got  many  ideas  about  how  to  better  manage  and  assess  our  REU.    Learning  best  practice  from  other  PIs,  meeting  NSF  program  directors,  build  Pan  REU  community.  I  was  the  most  impressed  by  the  fact  that  we  finished  with  action  plans,  rather  than  just  talking  about  issues.  Thanks  to  all  of  you  for  a  great  agenda.  The  REU  community.  The  PIs,  Program  Officers,  and  others  are  so  open  to  sharing  and  truly  collaborative;  it's  a  model  of  how  science  is  really  supposed  to  work  together  to  advance  a  greater  goal.  I  learned  so  much  about  the  approaches  being  used  by  other  programs,  to  achieve  the  goals  of  the  REU.    It  was  very  informative.  meetings  with  NSF  officers  Able  to  hear  about  best  practices  from  other  PIs.  Always  learn  something  new  at  these  meetings.  The  small  group  working  sessions  Interactions  with  the  program  officers  and  with  each  other  Learning  about  ways  other  REU  sites  handle  things.  Networking  with  other  PIs  and  program  officers  Helping  students  build  careers,  one  of  the  groups  in  which  I  participated.  It  was  good  to  connect  with  PIs  from  other  disciplines  and  learn  from  their  experience.  Connecting  with  others  Networking  with  other  REU  PIs  especially  those  from  other  disciplines  and  NSF  POs  Listening  to  other  tactics  to  broaden  participation.  I  also  think  that  it  codified  that  practices  used  in  some  areas  can  be  expanded;  specifically  common  application  deadlines  and  the  ability  to  share  applicant  pools  (with  applicant  permission).    Networking  with  other  REU  program  directors.  Discussions  about  common  portal  and  longitudinal  tracking  Networking  with  REU  PIs  from  across  disciplines/across  the  nation;  sharing  best  practices  The  gathering  of  so  many  focused,  energetic,  passionate  people  allowed  us  to  get  so  much  done  towards  creating  some  initial  pan-­‐NSF  REU  work.    It  was  so  valuable  to  find  out  what  other  disciplines  were  doing  with  their  REU  Leadership  Groups  and  to  learn  about  their  websites  and  other  resources.      Meeting  with  program  officers.    Opportunity  for  group  to  think  more  broadly  about  communicating  the  value  of  the  program.  Interacting  with  other  REU  PI's.  I  liked  that  different  disciplines  were  present.  Sharing  with  more  experienced  REU  PI.  I  learned  a  lot  of  new  information!    Seeing  the  variety  of  programs  to  learn  about  similar  efforts  in  the  other  divisions  Learned  various  program  execution  activities  from  others  to  REU  program  Evaluation  and  assessment  Broadening  participants    Networking  and  sharing  best  practices  Learning  about  other  sites  and  disciplines.  Information  sharing,  connecting  with  PIs  from  other  NSF  directories,  communication  with  NSF  program  directors  The  workshop  on  communicating  with  stakeholders  about  metrics  by  which  successful  REU  programs  should  be  measured.  Having  NSF  and  the  PI  on  the  same  room  sharing  ideas  and  visions.  

 

54  

Networking  hearing  best  practices  and  insights  from  other  PIs.    Thinking  about  topics  I  have  not  considered  before  just  running  my  own  REU,  it  was  great  to  hear  about  the  bigger  picture  and  what  a  panREU  group  can  accomplish.  Learning  about  practices  from  other  programs,  such  as:  (i)  methods  for  recruiting  students  from  underrepresented  groups  (ii)  connecting  with  faculty  and  students  from  community  colleges  (iii)  tracking  students  A  few  new  ideas,  mostly  meeting  new  colleagues.  Networking  with  other  PIs  and  learning  about  best  practices  The  small  working  groups  to  make  progress  on  topics/areas  of  interest.    Ideas  on  how  to  improve  my  individual  program.  Learning  about  the  diversity  of  REU  programs  offered.  making  personal  connections  with  NSF  and  REU  colleagues  The  collaboration  with  REUs  from  other  directorates  Meeting  my  REU  colleagues.  Identifying  common  challenges  across  sites  and  brainstorming  solutions  for  them.      talking  with  colleagues  Action  items  that  resulted  from  breakout  discussions  Networking  with  other  PIs  getting  ideas  about  what  others  are  doing    Learning  about  issues  that  some  or  all  REU  sites  face.  Community    4.  Which  of  the  following  are  true  regarding  your  experience  at  the  workshop?  

Answer  Options  Response  Percent   Response  Count  

I  reconnected  with  old  colleagues   61.7%   37  I  met  new  colleagues   93.3%   56  I  learned  new  strategies  for  running  an  REU   80.0%   48  I  learned  more  about  the  broader  REU  community   96.7%   58  Other  (please  specify)   4    

   

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

I reconnected with old colleagues

I met new colleagues

I learned new strategies for

running an REU

I learned more about the broader REU community

Which of the following are true regarding your experience at the workshop?

 

55  

5.  Please  rate  the  value  each  component  of  the  workshop:  

Answer  Options   Not  valuable   Valuable   Very  

valuable  

Did  not  attend  /  Unsure  

Rating  Average  

Response  Count  

Poster  session  at  NSF   6   28   11   15   3.58   60  Panel  of  NSF  Program  Officers   3   24   31   0   3.48   58  Best  practice  breakout  groups  &  reporting  out  on  Friday  morning   0   21   38   0   3.64   59  

Presentations  on  REU  communities  in  GEO,  CHM,  BIO,  AST   6   30   21   2   3.32   59  

REU  common  assessment  tool  presentation  and  breakout  session  on    

10   30   20   0   3.17   60  

Action  planning  breakout  sessions  &  presentations  

2   17   40   1   3.67   60  

Networking  at  meals,  breaks,  and  after-­‐hours   0   21   38   1   3.67   60  

Other  (please  specify)   2      

   6.  Which  Action  Plan  group  did  you  participate  in?  

Answer  Options   Response  Percent  

Response  Count  

Developing  collaborations  with  MSIs   15.5%   9  Developing  collaborations  with  2-­‐year  colleges   12.1%   7  The  development  of  a  Pan  REU  website   5.2%   3  Informing  the  development  of  long  term  tracking  tools   15.5%   9  Formalizing  the  Pan  REU  community   5.2%   3  

2.80 3.00 3.20 3.40 3.60 3.80

Poster session at NSF

Best practice breakout groups & reporting out on Friday morning

REU common assessment tool presentation and breakout session on

Networking at meals, breaks, and after-hours

Please rate the value each component of the workshop:

 

56  

Communicating  on  evaluation  criteria  and  metrics  for  success  with  REU  stakeholders   8.6%   5  Strategies  for  broadening  diversity   20.7%   12  How  can  the  REU  group  inform  the  agenda  and  priorities  for  congress?   13.8%   8  Did  not  attend  /  Not  applicable   3.4%   2    

   7.  How  would  you  rate  the  value  of  the  Action  Plan  breakout  group  session  that  you  participated  in?  

Answer  Options   Response  Percent  

Response  Count  

Not  valuable   1.7%   1  Valuable   35.0%   21  Very  valuable   58.3%   35  Unsure/NA   5.0%   3    

   8.  

Which Action Plan group did you participate in? Developing collaborations with MSIs

Developing collaborations with 2-year colleges

The development of a Pan REU website

Informing the development of long term tracking tools

Formalizing the Pan REU community

Communicating on evaluation criteria and metrics for success with REU stakeholders Strategies for broadening diversity

How can the REU group inform the agenda and priorities for congress?

How would you rate the value of the Action Plan breakout group session that you participated in?

Not valuable Valuable

Very valuable Unsure/NA

 

57  

What  are  you  thoughts  that  Action  Planning  topic  that  you  discussed  in  the  breakout  group?  

Answer  Options   Response  Count  

    36    Response  Text  

It  seems  to  me  that  there  are  existing  strategies  for  tracking  a  student  that  signs  up  for  a  program  using  a  simple  log-­‐in.    See  NASA  OSSI.    That  requires  a  common  initial  (demographic)  application  and  then  programs  could  ask  specific  questions.    NSF  is  asking  for  a  lot  of  input,  which  is  fine,  but  I  would  also  like  to  see  more  leadership  decision-­‐making  on  these  points.    If  the  solicitation  says  that  PIs  will  use  a  common  application,  then  everyone  will  (like  BIO  using  a  common  assessment  tool)  -­‐  mandate  it  instead  of  having  each  us  fund  the  expense  of  designing  tracking  tools.  It  is  my  personal  belief  that  community  colleges  are  an  untapped  resource.  I  am  happy  to  see  the  interest  in  NSF  and  my  colleges  in  engaging  with  community  colleges  more  establishing  a  reu  msi  leadership  group  Strategizing  on  how  to  make  the  REU  outcomes  more  visible  to  congress.  A  central  website  seemed  to  be  a  vehicle  that  could  solve  many  issues  raised  at  this  meeting  -­‐  from  helping  PIs  with  information  to  publicizing  great  outcomes.  I  met  some  outstanding  organizers  and  made  lots  of  nice  connections.  We  do  need  to  meet  in  person  like  this  from  time  to  time  to  seriously  improve  our  programs  and  increase  quality.    We  made  progress  in  crystallizing  a  common  vision  for  REU  program  outcomes,  with  important  caveats  and  conditions.  We  identified  one  fairly  concrete  application  of  what  we  discussed  -­‐  to  provide  input  for  the  new  solicitation.  We  also  identified  the  need  to  work  with  Mathematica  and  others  working  on  common  assessments  and  a  common  voice  of  program  success  (a  bit  more  vague,  perhaps).  However,  we  probably  could  have  come  up  with  some  more  concrete  strategies  for  communicating  the  importance  of  a  diversity  of  outcomes  both  within  and  among  programs  to  key  stakeholders  such  as  PI's,  mentors,  host  institutions,  etc.  It  was  very  informative  It's  important  to  have  NSF  PDs  join  the  conversation  to  understand  what  REU  community  can  do  to  showcase  the  evidence-­‐based  support  to  congress  about  the  importance  of  the  program.    Not  sure  of  the  question...  It  was  hard  to  choose  an  Action  Plan  topic,  because  all  of  the  topics  were  important  ones.  I  thought  the  one  I  chose  was  very  useful,  as  we  considered  the  broader  scale  ways  to  promote  and  support  a  Pan-­‐REU  commuinity  overall.  Good  mix  of  types  of  REU  programs.  Helpful  to  have  former  review  panel  members  describe  metrics  on  grant  reviews  to  inform  our  discussion.  The  size  of  the  group  that  I  participated  in  was  great.    We  had  a  diverse  number  of  institutions  and  it  worked  really  well.  Including  our  program,  there  appear  to  be  a  number  of  dedicated  PIs  who  work  hard  to  find  a  bit  of  funding  to  improve  their  undergraduate  programs.  For  example,  a  few  of  them  find  money  to  bring  in  professors  from  their  students'  home  institutions,  especially  if  these  institutions  do  not  have  a  research  program.  It  appears  that  the  funds  are  meager  and  can  only  support,  for  example,  travel.  This  is  a  huge  barrier.  It  requires  the  institutions  conducting  REU  programs  are  endowed  with  resources.  This  automatically  creates  a  disparity  among  REU  programs.  It  is  highly  recommended  that  NSF  allow  some  money  for  a  professor  from  REU  student's  home  institution  to  participate  along  with  the  student.  At  high  school  level,  there  are  individual  programs  (non-­‐NSF)  where  a  teacher  and  student  come  as  a  team  and  do  summer  research.  This  may  be  a  good  model  to  adopt.  A  small  pilot  program  would  be  well.  Many  of  the  groups  share  common  goals  so  connecting  them  is  key  to  avoid  duplicate  work.  Need  to  form  a  cross-­‐disciplinary  leadership  group  The  tactics  used  by  individual  groups  are  less  important  than  encouraging  each  PI  and  their  colleagues  to  strategize  and  keep  reinventing  their  strategies.  One  size  can  never  fit  all.    Not  sure  what  you  are  asking...  Efforts  to  develop  the  pan-­‐NSF  REU  web  page  including  planning  the  layout  and  content  and  other  related  pan-­‐community  activities  will  be  needed.    This  is  true  for  the  action  plan  items  that  came  out  of  this  workshop.  

 

58  

It  will  be  valuable  to  create  a  pan  website  and  LG  but  it  will  be  difficult  to  keep  the  momentum  going  once  we  disperse.    I  am  very  interesting  in  increasing  participation  in  the  REU  program  and  attracting  a  diverse  group  of  students.  I  think  we  left  with  a  concrete  plan  to  do  so.  A  lot  of  time  was  spent  talking  about  what  we  were  already  doing.  -­‐  Active  participation  in  the  leadership  council  in  the  topic  area  (collaborating  with  2-­‐year  colleges)  -­‐  Developing  strong  partnership  with  community  colleges  in  GA  and  beyond  I  thought  it  was  an  interesting  topic  and  the  participants  gave  good  input.  Good  start  to  plan  actions  to  address  common  REU  concerns,  develop  useful  tools  that  take  load  off  from  PIs  and  REU  participants,  awareness  of  resources  available  and  plans  to  aggregate  best  practices  and  broad  dissemination  to  the  larger  REU  community.    The  conversations  about  the  the  different  ways  in  which  success  could  be  measured  was  extremely  valuable.  It  made  me  think  more  deeply  about  considering  the  purpose  of  individual  programs  in  assessing  success.  It  is  critical  to  ensure  that  funds  are  available  NSF  to  support  REUs  programs.    This  program  has  an  incredible  impact  by  providing  students  with  learning  experiences  beyond  classes.    Providing  research  experience  is  one  of  the  very  few  healthy  educational  activities  has  remained  in  our  current  education  system.  pilot  to  give  REU  applicants  NSF  ID.    I  was  hesitant  at  first,  but  group  made  good  arguments  why  this  is  needed  and  how  we  can  implement  without  too  much  burden  on  PIs  and  students.  Success  in  this  area  involves  connections  between  community  colleges  and  four-­‐year  colleges  and  universities.  This  relationship  would  provide  research  opportunities  for  both  faculty  and  students  at  community  colleges.  That  it  is  important  to  explore  this  issue  further...  especially  from  the  NSF  perspective,  as  well  as  from  a  more  organic  PI  driven  approach  I'm  excited  about  the  energized  group  we  had  and  specific  plans  we  set.  Formation  of  REU2N  and  already-­‐active  leadership  and  NSF  connections  -­‐  very  productive  It's  always  effective  to  develop  an  Action  Plan,  but  there  needs  to  be  follow  up  to  ensure  the  steps  are  taken,  i.e.,  people  are  held  accountable.  Including  Community  College  Faculty  is  a  vital  part  of  the  process.  REU  participants  are  excellent  ambassadors  and  role  models.  Doing  this  work  *properly*  means  expanding  beyond  the  current  scope  of  REUs,  which  inevitably  means  requiring  more  funding.    needs  further  refinement  Advertising  the  REU  program  more  broadly  at  NSF  can  help  to  broaden  participation  across  the  community.    We  need  long-­‐term  follow  up  with  REU  students  and  mentors  to  broaden  diversity  in  the  faculty  and  scientific  community  as  a    whole.    :-­‐)    9.  How  helpful  would  these  tools  be  for  sharing  resources  and  strategies  with  PIs  across  the  disciplines:  

Answer  Options   Poor   Good   Very  good  

Unsure/NA  

Rating  Average  

Response  Count  

A  pan-­‐NSF  REU  web  page   1   7   50   1   3.03   59  An  email  listserv  for  REU  PIs  across  disciplines   4   17   36   2   3.03   59  Telecons  on  relevant  topics   5   27   23   4   3.12   59  Sessions  (oral,  poster)  at  national  science  meetings  

6   30   21   2   3.00   59  

Workshops  at  national  science  meetings   3   24   30   2   3.05   59  A  future  pan-­‐NSF  REU  meeting  in  ~2  years   0   9   48   2   3.10   59  Please  elaborate,  or  suggest  other  methods.   18    

Please  elaborate,  or  suggest  other  methods.  

 

59  

 

   10.  

2.90 2.95 3.00 3.05 3.10 3.15

A pan-NSF REU web page

An email listserv for REU PIs across

Telecons on relevant topics

Sessions (oral, poster) at national science

Workshops at national science meetings

A future pan-NSF REU meeting in ~2

How helpful would these tools be for sharing resources and strategies with PIs across the disciplines:

"A  future  pan-­‐NSF  REU  meeting"  in  4-­‐5  years  The  community  needs  more  time  and  resources  to  carry  out  many  of  the  terrific  suggestions  that  came  up  at  the  meeting.  I  am  doubtful  about  the  benefit  of  teleconferences:  It  is  much  easier  to  make  real  progress  in  person.    I'm  not  sure  we  need  another  pan  REU  meeting  in  2  years  ...  unless  there  are  specific  initiatives  that  developed  out  of  this  meeting  that  would  really  benefit  from  a  meeting  that  soon.  Otherwise,  perhaps  in  5  years?  My  difficulty  with  workshops  and  sessions  at  national  meetings  is  that  I  am  already  participating  very  heavily  at  those  meetings  in  other  workshops  and  sessions.    I  would  make  the  time  for  an  REU  session,  but  I  like  having  it  be  the  only  thing  that  I  am  focusing  on.    So,  the  REU  PI  and  Pan  REU  meetings  were  nice.  I  would  substitute  webinars  for  telecons,  and  then  I  would  assign  "very  good".  The  reason  I  gave  "poor"  rankings  to  the  national  science  meetings  is  that  very  few  of  us  go  to  meetings  that  are  outside  of  our  own  discipline,  and  it  would  be  difficult  to  get  the  funds  to  do  so.  I  like  the  meetings  but  not  five  years  between  when  we  get  together.    The  time  of  year  is  also  hard  (graduation  and  finals)  Perhaps  3-­‐5  years  before  the  next  pan  workshop  A  pan  NSF  web  page  is  good,  if  it  is  organized  along  disciplines  or  interconnected  disciplines.  Private  Facebook  group  as  well  Pan-­‐NSF  meetings  should  not  be  too  large.    This  size  was  perfect.  Meeting  face-­‐to-­‐face  was  very  valuable.  Create  taskforce  and  workgroups  and  make  resources  (funding,  support)  available  to  tackle  the  issues    I  think  there  can  be  email  overload  at  times.  Having  a  site  where  PI's  can  find  resources  when  there  is  a  need  would  be  most  helpful.    The  strategies  that  allow  every  PI  to  learn  and  participate  is  best,  but  the  leadership  meetings  are  important  also.  I  prefer  videocons  (we've  used  zoom.us  for  large  groups  of  100+  participants  with  no  trouble)  as  it  helps  improve  relationships    Every  2  years  is  probably  too  frequent  I  would  suggest  pan  REU  meetings  every  5  years  (2  yrs  too  short,  10  years  too  long)  

 

60  

Please  rate  these  aspects  of  the  workshop  planning  

Answer  Options   Poor   Good   Very  good  

Unsure/NA   Rating  Average  

Response  Count  

Travel  and  hotel  reservations   0   8   50   2   3.93   60  Online  poster  abstract  &  poster  submission  process  

1   12   23   23   4.53   59  

Pre-­‐workshop  communication  and  service   2   11   46   1   3.75   60  On-­‐site  registration/check-­‐in   0   4   55   1   3.97   60    

   11.  Please  rate  the  conference  facilities  and  meals  

Answer  Options   Poor   Good   Very  good  

Unsure/NA   Rating  Average  

Response  Count  

Rosslyn  ballroom   2   22   36   0   2.57   60  Breakout  rooms   1   28   31   0   2.50   60  Hotel  guest  room   3   24   29   3   2.54   59  Meals   6   34   19   0   2.22   59    

0.00 1.00 2.00 3.00 4.00 5.00

Travel and hotel reservations

Online poster abstract & poster submission

process

Pre-workshop communication and

service

On-site registration/check-in

Please rate these aspects of the workshop planning

 

61  

   12.  What  suggestions  do  you  have  to  improve  the  workshop,  either  in  terms  of  format  or  content  covered?  

Answer  Options   Response  Count  

    27    Response  Text  

More  notice  on  the  option  to  prepare  a  poster  and  having  the  guest  stay  in  a  hotel  that  is  closer  to  NSF,  if  possible,  would  have  been  great  Networking  with  PIs  from  other  directorates  It  would  have  been  nice  to  hear  a  few  REU  stories,  individual  site  statistics,  international  REU  successes,  and  presentations  from  new  REU  PIs  about  their  upcoming  programs.  Program  officers  were  not  as  clear  or  informative  as  they  could  have  been.  The  vague  outcomes  are  not  helpful.  It  would  have  been  nice  to  have  seen  some  data  from  the  NSF  side  on  the  impact  of  these  programs  nation  wide.  I  found  almost  every  aspect  valuable,  but  I  did  not  like  the  one-­‐size-­‐fits-­‐all  approach  to  a  common  application.  Each  area  needs  to  be  aware  that  other  REU  areas  are  very  distinct,  and  have  their  own  dynamic.  This  is  why  it  is  valuable  we  meet  in  person  and  learn  from  each  other.    I  think  it  was  great!  Perhaps  a  bit  more  of  a  profile  about  the  participants  that  we'd  get  a  week  before  (or  even  at  registration)...  maybe  based  on  a  survey  we'd  complete  a  month  or  more  ahead  of  time,  categorizing  our  REU  site  program  and  our  own  expertise  in  a  number  of  ways.  So  we'd  know  things  like  multi-­‐disciplinary,  distributed  site,  international;  basic  mentoring  model;  targeted  student  population;  whether  we  could  offer  specific  help  in  things  like  recruitment,  evaluation,  research,  mentor  support,  etc.  I  like  this  format.    It  works  well  for  me.  I  liked  that  you  let  us  vote  on  what  topics  we  thought  were  most  important.  At  this  point  I  can't  remember  whether  you  gave  us  the  option  to  contribute  possible  topics  of  discussion.  The  size  is  good,  provides  an  opportunity  to  meet  almost  everyone.    Need  more  time  to  appreciate  all  the  posters.    Since  people  put  so  much  effort  into  it,  perhaps  3  minute  lightning  talks  from  all  posters  will  give  us  a  chance  to  hear  unique  elements  of  each  program.  Didn't  get  to  see  all  the  posters  in  my  area  Commend  the  organization  of  the  workshop.  As  a  participant,  I  saw  it  operate  flawlessly.  The  last  session  with  actions  could  be  elaborated.  5  mins  reports  are  too  short.    The  Q&A  after  the  reports  and  the  informal  panel  discussion  was  good.  May  be  the  panel  session  could  be  formalized  and  longer.  Since  a  participant  who  is  interested  in  multiple  topics  can  only  participate  in  one,  a  formal  pane  discussion  may  be  a  good  thing.  

2.00 2.10 2.20 2.30 2.40 2.50 2.60

Rosslyn ballroom

Breakout rooms

Hotel guest room

Meals

Please rate the conference facilities and meals

 

62  

Clearer  goals  from  the  beginning  and  a  clearer  connection  to  how  this  work  impacts  Congress  and  NSF  leadership.  ..this  was  an  excellent  meeting  It  would  be  helpful  to  have  some  more  food  options  for  those  with  dietary  restrictions.  This  was  a  great  meeting.    The  hotel  room  was  okay  and  probably  economical.  However,  it  was  very  very  loud  due  to  the  traffic.    I  thought  it  went  well.    I  would  have  liked  more  time  to  discuss  but  I  also  realize  that  people  are  busy  and  that  it  would  cost  more.  Breakout  sessions  are  very  useful.    The  next  workshop  can  have  a  session  dedicated  to  outcomes  of  the  action  plans  that  are  executed  as  a  result  of  this  meeting.  These  sessions  could  be  used  as  a  means  of  sharing  best  practices.  Include  very  short  presentation  by  a  few  PIs  to  discuss  proven  successful  practices.    All  excellent!  Inclusion  of  a  conference  proceedings  My  only  complaint  was  the  food.  As  a  person  with  dietary  restrictions,  the  options  were  abysmal.    Knowing  that  the  poster  session  would  be  held  at  NSF  in  advance  would  have  been  appreciated.  My  flight  was  scheduled  to  land  right  at  4pm,  which  means  I  missed  the  first  half  of  the  poster  session.  This  wasn't  communicated  in  advance,  and  when  I  checked  whether  a  later  flight  would  be  ok,  I  was  told  there  was  no  plans  for  Thursday  beyond  check  in.    next  time  I  would  suggest  pre-­‐assigning  reporters.  A  bit  more  pre-­‐communication  and  specific  homework  prior  to  the  workshop  would  help.    Invite  more  new  REU  sites  or  potential  REU  applicants  to  participate  in  the  workshop.    The  information  circulated  would  be  extremely  valuable  to  these  cohorts.    13.  Do  you  have  any  other  comments  or  questions?  

Answer  Options   Response  Count  

    20    Response  Text  

The  PAN  REU  workshop  should  take  place  every  5  year  instead  of  10  years  Hot  food  for  breakfast  and  lunch  would  have  been  preferred.  I  do  think  regular  meetings  would  provide  a  venue  for  getting  some  traction  on  some  of  the  issues  raised.  The  REU  programs  have  a  lot  of  commonalities  so  working  collectively  could  have  a  big  impact.  Thanks  so  much  for  all  the  effort  you  put  into  the  workshop.  No  The  organizers  have  done  a  wonderful  job  putting  this  workshop  together.      Thanks  again  for  all  your  hard  work.  Great  job.  NSF  does  an  excellent  job  with  the  REU  program.  As  educators,  we  have  to  find  effective  ways  to  convince  our  representatives  that  this  is  a  splendid  organization  and  a  great  investment  in  the  future  of  the  nation.  This  was  a  great  beginning  to  a  challenging  and  needed  conversation.  Hopefully  the  momentum  continues.  Thank  you  so  much  for  an  incredible  workshop,  and  thanks  to  NSF's  Program  Officers  for  coming.    There  was  a  lot  of  work  by  the  organizers.  Thank  you.    I  think  that  we  need  a  REU  handbook  for  new  PI's.  That  would  be  a  nice  resource  to  develop.  Just  appreciation  for  everyone  involved  in  putting  it  together.  Would  be  nice  if  the  report  can  be  disseminated  to  all  attendee.    Overall,  this  was  very  good  exposure  and  information.  I  hope  I  can  contribute  realization  of  some  of  the  ideas  that  

 

63  

were  proposed  on  Saturday.  This  workshop  was  well  organized.  Just  want  to  have  strong  follow  up  very  soon  after  the  meeting  so  we  can  share  resources  and  don't  lose  momentum.      This  was  a  great  workshop  with  concrete  outcomes,  such  as  starting  action  items  to  improve  assessment,  participant  tracking,  and  furthering  the  Pan-­‐REU's  community  connection  with  community  colleges.  I  like  the  interactive  format  and  sharing.  Thanks  to  the  organizers.    Please  post  all  presentation  on  the  website  for  follow  up.      14.  Which  NSF  Division  funds  your  REU?  

Answer  Options  Response  Percent  

Response  Count  

BIO   25.4%   15  CISE   15.3%   9  CISE/ACI   0.0%   0  STC     0.0%   0  EHR   3.4%   2  ENG   6.8%   4  GEO   5.1%   3  GEO/AGS   1.7%   1  GEO/EAR   1.7%   1  GEO/OCE   3.4%   2  MPS/AST   1.7%   1  MPS/CHE   11.9%   7  MPS/DMR   3.4%   2  MPS/DMS   13.6%   8  MPS/PHY   10.2%   6  SBE   3.4%   2  Other  (please  specify)   4    Other  (please  specify)  

NSF  staff  thanks  for  all  of  you  hard  work  in  organizing  this  meeting!    Used  to  be  MPS/DMR  and  will  be  ENG  beginning  this  year.  Thank  you!      

 

64  

   

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

Which NSF Division funds your REU?