QEP Survey #3 Findings

April 10th, 2015

Prepared by:

Jaime B. Henning, Ph.D.

Jean-Paul Philippe, I-O Psychology MS Student

Dr. Jaime Henning – Primary Contact jaime.henning@eku.edu

SUMMARY OF METHODOLOGY AND RESULTS

In February 2015, an online survey was sent to department chairs and program coordinators in order to gather program-level evidence for student performance in three areas identified via an earlier survey as potential QEP topics (Critical Reading Skills, Information Literacy, and Applying Discipline Specific Concepts). Department chairs were instructed to seek faculty input to complete the survey. Data were provided from 47 program representatives. Of these participants, the majority (76.6%) represented undergraduate programs.

When asked about students’ critical reading skills, 36.1% of respondents indicated that students’ general reading skills are “Good” or “Very Good,” while 34.5% of respondents indicated that students’ reading skills are “Weak” or “Extremely Weak.” Additionally, 31.9% of respondents indicated that students’ discipline-specific readings skills are “Good” or “Very Good,” while 38.3% of respondents indicated that students’ discipline-specific reading skills are “Weak” or “Extremely Weak.” Furthermore, when respondents were asked to indicate sources of evidence used to support their opinions regarding students’ critical reading skills, the majority of participants indicated relying on faculty observation and the evaluation of student work, while far fewer reported relying on standardized or professional exams, employer/practicum supervisor observations, and student perceptions of their own skills.

Participants were also asked to evaluate students on general information literacy skills. Of the 47 respondents, 21.3% indicated that students are “Good” or “Very Good” with regard to these skills, while 25.5% indicated students are “Weak” in this area. Additionally, when asked about discipline-specific information literacy skills, 51.1% of respondents indicated students are “Ok” in regard to these skills, while 21.3% rated students as “Weak” or “Extremely Weak.” Furthermore, the majority of respondents indicated relying on faculty observation and evaluation of student work to support their opinions, with few relying on standardized or professional exams, employer/practicum observations, and student perceptions of their own skills.

Finally, respondents were asked to rate students’ ability to apply discipline-specific concepts to issues of importance to their field. Of the 46 respondents to this questions, 40.5% indicated that students’ ability was either “Good” or “Very Good,” while only 12.8% indicated students’ ability as “Weak” or “Extremely Weak.” Furthermore, the majority of participants reported faculty observation and evaluation of student work as the types of evidence used to support their opinions.

Based on these results, it appears respondents more frequently rated students as “Weak” or “Extremely Weak” on Critical Reading Skills (34.5% for general skills, 38.3% for discipline-specific skills), followed by Information Literacy Skills (25.5% for general skills, 21.3% for discipline-specific skills), and Ability to Apply Discipline-Specific Concepts (12.8%). Furthermore, in all cases the majority of respondents indicated relying most frequently on faculty observation and evaluation of student work to provide evidence for their opinions. Far fewer respondents reported standardized or professional exams, employer/practicum observations, and student perceptions of their own skills as available forms of evidence.

Charts and comments are presented below to provide further information. These include greater detail regarding ratings of students’ skills and abilities on each of the three topics, and on the forms of evidence programs rely on to inform these evaluations.

Programs Represented

Academic Readiness & Eastern Bridge program

Academic Tutoring Agriculture Animal Studies Anthropology Program Department

of Anthropology, Sociology, and Social Work

Associate Degree Nursing Associate of General Studies B.S. Psychology BA Geography BGS: College Tracks Biology BS Biology MS Biology Teaching BS BS Earth Science Teaching BS Geology Clinical Psychology CMS (Communication Studies) College of Business and

Technology - Marketing Department

Corporate Communication and Technology

Economics Education Pays EKU Honors Program EKU Libraries English 101/102 (First-Year Writing)

Environmental Health Science Environmental Studies BS Fly General Psychology Masters

Program Globalization and International

Affairs (GLO) Government -- Paralegal, Political

Science, Public Administration History and History Teaching I-O Psychology MS Program Journalism Justice Studies McNair Scholars Program Medical Laboratory Science Noel Studio for Academic Creativity NOVA Student Support Services

Trio Program Pre-Law Pre-Professional Advising Pre-Professional Advisor for Health

Programs Professional Track of BGS: (Adult

Completer Degree) Public Relations Safety, Security, Emergency

Management Social Work Sociology Teaching & Learning Center Wildlife Management BS

TOPIC #1: CRITICAL READING SKILLS

Other

Standardized/Professional Exams

Employer/Practicum Supervisor Observations

Student Perception of Their Own Skills

Evaluation of Student Work

Faculty Observation

0 2 4 6 8 10 12 14 16

5

7

2

10

15

14

Forms of Evidence Indicated by Respondents who Rated Students' Reading Skills as Weak or

Extremely Weak

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Reading Skills as Weak or Extremely Weak are reported as frequencies.

Other

Standardized/Professional Exams

Employer/Practicum Supervisor Observations

Student Perception of Their Own Skills

Evaluation of Student Work

Faculty Observation

0 2 4 6 8 10 12 14 16

1

5

5

2

15

14

Forms of Evidence Indicated by Respondents who Rated Students' Reading Skills as Good or Very

Good

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Reading Skills as Good or Very Good are reported as frequencies.

Other Evidences Indicated by Respondents who rated students’ Reading Skills as:

Very Weak:

Tutors: Students are required to attend study sessions with tutors. Tutors report that poor reading skills affects almost every assignment.

Consultation records from Writing Fellows program

Weak:

Some students writing theses can't seem to figure out what is important in what they read, and don't seem to know how to integrate what they read into a coherent introduction.

We see their writing in reaction to what they’ve read, which exhibits they often may have misread the text, misunderstood the author’s motives, may not be able to identify main points or thesis statements, or find college-level reading too difficult. They often have not been taught to annotate their reading or practiced that until they get to our classes, and then they may have difficulty making that a common practice. They often have even more difficulty reading something they potentially disagree with, finding it difficult to separate their own opinions on the topic from analyzing the work at hand.

These responses are based on feedback Lara Vance has received from tutors: Reviewing student notes. Dialogue during tutoring sessions. Review of notes/grading comments made by instructors. Since receiving instruction in how to recognize a reading issue (from Dr. Lisa Bosley), more tutors recognize when they are working with someone who has difficulty reading (comprehension, vocabulary, speed, etc.).

Evaluation of note taking skills. Evaluation of classroom participation.

Ok:

Data are scattered among different departments that teach required majors courses. I’ve thought about how to address the questions and I honestly have to respond that I do not know how to answer the questions in a why in which I can provide evidence to support my assessment. For example, of the 25-26 classes required in the Land Resources Option (range of numbers because the student may take one or two math classes), Biology teaches 6 of the classes (BIO 111, 112, 316, 320, 495, 521 or 536)….the rest of the courses, about 75%, are distributed among 8 departments (CHE, GLY, GEO, EHS, PHI, STA, MAT, AGR)…and if the student doesn’t do a coop (BIO 349), the required independent study class will be taken in CHE, ENV, GEO, or GLY. I have no idea how faculty in the other programs would rate the students in terms of reading, literacy, or ability to apply discipline-specific concepts. In my opinion, the best class in which to assess this material would be in the required independent study class….but to my knowledge, Biology has never received any feedback regarding these students from the other programs. The same general trend applies to the Natural History Option. I’m not sure how to answer the QEP survey’s questions for a degree program that was designed to be interdisciplinary…maybe Rose Perrine can answer that?? Chuck Elliott

Advising.

Observations in the library instruction classroom, while answering reference questions, and during research appointments

Our assessment is of skills achievement in the areas of critical thinking and written and oral communication, particularly analytic essays. Therefore each year we collect reading response papers, primary source analysis essays, and research papers at various levels in our program to assess our students achievement. PLEASE NOTE THAT THIS SURVEY WILL NOT ALLOW FOR A DIVISION AMONG FACULTY. ON QUESTIONS 3 AND 4 FACULTY EVENLY SPLIT BETWEEN OK AND GOOD. FACULTY IN MAKING THESE RATINGS NOT THAT BECAUSE WE ARE A READING AND WRITING INTENSIVE DISCIPLINE WE DO NOT ATTRACT THOSE WITH VERY WEAK SKILLS. OUR MAJORS TEND TO COME IN WEAK AND BECOME OK or COME TO OUR MAJOR WITH OK SKILLS AND PROGRESS TO GOOD SKILLS.

Good:

Students in our major preform adequately to quite well on current events quizzes, when they choose to read. The choosing to read is the variable that provides the evidence. Students frequently admit to not reading when they have a low score. Rarely if ever do students state they did not understand the news stories assigned once they take time to read them.

Reading course examinations

We have many opportunities in our courses to observe students ability to synthesize and analyze assigned reading materials which suggest good (not very good) read skills for our students

Semester academic progress reports from participants' instructors.

Very Good:

Most of our graduate students have scored over 460 on the verbal portion of the Graduate Record Exam. That score places students at 51st percentile, meaning that, among all those who take the exam, our graduate students rank in the top 50% (and most much higher than that). As such, even when entering our program, our graduate students have demonstrated competence in their reading and verbal reasoning skills. In addition, all of our graduate-level courses require significant amounts of reading (of texts as well as, in some classes, the primary scientific literature) and students would not be able to do well in those classes without excellent reading skills. Finally, all of our graduate students must prepare and research proposal and, prior to graduating, a thesis. Doing so requires that students are very familiar with the primary literature which necessary involves much reading.

TOPIC #2: INFORMATION LITERACY

Other

Standardized/Professional Exams

Employer/Practicum Supervisor Observations

Student Perception of Their Own Skills

Evaluation of Student Work

Faculty Observation

0 2 4 6 8 10 12

2

2

2

4

10

9

Forms of Evidence Indicated by Respondents who Rated Students' Information Literacy Skills

as Weak or Extremely Weak

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Information Literacy Skills as Weak or Extremely Weak are reported as frequencies.

Other

Standardized/Professional Exams

Employer/Practicum Supervisor Observations

Student Perception of Their Own Skills

Evaluation of Student Work

Faculty Observation

0 2 4 6 8 10 12

2

2

2

2

11

11

Forms of Evidence Indicated by Respondents who Rated Students' Information Literacy Skills as

Good or Very Good

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Information Literacy Skills as Good or Very Good are reported as frequencies.

Other Evidences Indicated by Respondents who rated students’ Information Literacy skills as:

Very Weak:

Our assessment for the past two cycles has shown that students in our program struggle with integration and cohesion, which indicates a greater issue at hand with information literacy skills. They may be able to find information but then not know what to do with it once they have it. For example, they may appropriately insert a quote and documentation but not integrate smoothly. In the worst cases, they accidentally plagiarize because they struggle with intellectual property concepts requiring them to quote, paraphrase, and cite. Even in the best case scenario, they struggle with the idea that information should be used to enter into a conversation and build their own evidence for development; in other words, their own authorship gets overwhelmed with their research.

These responses are based on feedback Lara Vance has received from tutors: Reviewing student notes. Dialogue during tutoring sessions. Review of notes/grading comments made by instructors. Since receiving instruction in how to recognize a reading issue (from Dr. Lisa Bosley), more tutors recognize when they are working with someone who has difficulty reading (comprehension, vocabulary, speed, etc.).

Inability to apply information found in reading assignments to classroom activities.

Weak:

Advising.

Students struggle in a media-saturated environment to identify appropriate sources of material in first drafts for news stories and for media law and media research papers. In news classes and in research papers, much time must be devoted to dissecting source credibility so that the second drafts of stories are more in-line with discipline expectations. Weaker students flounder because they do not know where to begin to seek out information. They state they are accustomed to having information provided to them and do not know where to begin to seek out information independently. In response to this need, a faculty member is developing a course in information gathering.

Ok:

Our assessment is of skills achievement in the areas of critical thinking and written and oral communication, particularly analytic essays. Therefore each year we collect reading response papers, primary source analysis essays, and research papers at various levels in our program to assess our students achievement. PLEASE NOTE THAT THIS SURVEY WILL NOT ALLOW FOR A DIVISION AMONG FACULTY. ON QUESTIONS 3 AND 4 FACULTY EVENLY SPLIT BETWEEN OK AND GOOD. FACULTY IN MAKING THESE RATINGS NOT THAT BECAUSE WE ARE A READING AND WRITING INTENSIVE DISCIPLINE WE DO NOT ATTRACT THOSE WITH VERY WEAK SKILLS. OUR MAJORS TEND TO COME IN WEAK AND BECOME OK or COME TO OUR MAJOR WITH OK SKILLS AND PROGRESS TO GOOD SKILLS.

Some students had difficulty finding materials to use in writing their thesis work, even though the materials were easily available.

Tutors' reports regarding the areas in which they are most asked for help.

Ability to critically analyze professional writing; verbal communications

Term paper and other short research exercises suggest that students are only okay in both their general and discipline-specific literacy skills.

I should qualify that generalizations of this sort would depend upon the level of the student in the Honors Program. In Honors Rhetoric (first-year honors students), reading comprehension skills and information literacy skills would be rated by our faculty as "Developing." Most of our honors thesis mentors, however, would regard their thesis mentees' skills in disciplinary reading comprehension and information literacy as "good" or even "very good," particularly by the end of the thesis process.

Good:

2015 Program Review data

Very Good:

Expanding on our response to question 1C, we require all of our M. S. students to take BIO 801 – Scientific Writing and Literature. In this course, students are required to write research proposals and research papers (in scientific journal format), critique the writing of others, give oral presentations, and prepare a poster. The oral presentations and poster are presented and prepared as they would be at a scientific meeting. Many of our graduate students have little or no prior experience with

scientific writing and must quickly become reasonably adept at these skills to receive a passing grade. Although students may not be accustomed to having their writing and presentations critically evaluated, such critiques are an essential part of becoming better writers and presenters. Writing proposals and papers and giving presentations also require critical and creative thinking. Students must become familiar with relevant scientific papers and must be able to place their work within the context of previous work, explaining how their results relate to those published previously and what their results have added to what was known previously. As an indication of our effectiveness at improving the creative thinking and effective communication skills of our graduate students, 98 graduate students have taken BIO 801 since 2008 and 93 have earned either an ‘A’ (62, or 63.3%) or a ‘B’ (31, or 31.6%). Only five (5.1%) earned a grade of ‘C.’ These grades are one indication of the very good writing skills of our graduate students In addition, many, if not most, of our graduate courses require some writing in the form of essays or reviews and doing well in the courses requires, among other things, good writing skills. Also, all of our graduate students must write a research proposal and, ultimately, a thesis. These must be approved by their three committee members and such approval requires that a proposal and thesis be well written. Our high M. S. degree graduation rates (indicating successful completion of a proposal and thesis) provide further evidence of the very good writing (and reading) skills of our graduate students.

Semester academic progress reports from participants' instructors.

TOPIC #3: APPLICATION OF DISCIPLINE SPECIFIC CONCEPTS

Other

Standardized/Professional Exams

Employer/Practicum Supervisor Observations

Student Perception of Their Own Skills

Evaluation of Student Work

Faculty Observation

0 1 2 3 4 5 6 7

3

2

1

0

6

6

Forms of Evidence Indicated by Respondents who Rated Students' Ability to Apply Discipline-Spe-

cific Concepts as Weak or Extremely Weak

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Ability to Apply Discipline-Specific Concepts as Weak or Extremely Weak are reported as frequencies.

Other

Standardized/Professional Exams

Employer/Practicum Supervisor Observations

Student Perception of Their Own Skills

Evaluation of Student Work

Faculty Observation

0 2 4 6 8 10 12 14 16

2

5

10

6

15

15

Forms of Evidence Indicated by Respondents who Rated Students' Ability to Apply Discipline-

Specific Concepts as Good or Very Good

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Ability to Apply Discipline-Specific Concepts as Good or Very Good are reported as frequencies.

Other Evidences Indicated by Respondents who rated students’ ability to Apply Discipline Specific Concepts as:

Extremely Weak:

These responses are based on feedback Lara Vance has received from tutors: Reviewing student notes. Dialogue during tutoring sessions. Review of notes/grading comments made by instructors. Since receiving instruction in how to recognize a reading issue (from Dr. Lisa Bosley), more tutors recognize when they are working with someone who has difficulty reading (comprehension, vocabulary, speed, etc.). Also, feedback from students attending tutoring sessions. For example, students will say to tutors, "I don't know why I need to know this."

Most students in the Eastern Bridge program have not yet had the opportunity to experience discipline-specific concepts in particular major areas. However, they exhibit a lack of critical thinking, critical understanding, and critical reading skills.

Weak:

Assessment of courses.

Ok:

Advising. Assessment of courses.

Students who have succeeded in courses where they are expected to engage in higher-order thinking skills and who have been exposed to multiple worldviews tend to be successful in making discipline-specific content applications in Media Ethics and Media Law. Test scores indicate that students who have not successful completed previous courses where they were expected to do high levels of analysis simply do not fare well.

Our assessment is of skills achievement in the areas of critical thinking and written and oral communication, particularly analytic essays. Therefore each year we collect reading response papers, primary source analysis essays, and research papers at various levels in our program to assess our students achievement. PLEASE NOTE THAT THIS SURVEY WILL NOT ALLOW FOR A DIVISION AMONG FACULTY. ON QUESTIONS 3 AND 4 FACULTY EVENLY SPLIT BETWEEN OK AND GOOD. FACULTY IN MAKING THESE RATINGS NOT THAT BECAUSE WE ARE A READING AND WRITING INTENSIVE DISCIPLINE WE DO NOT ATTRACT THOSE WITH VERY WEAK SKILLS. OUR MAJORS TEND TO COME IN WEAK AND BECOME OK or COME TO OUR MAJOR WITH OK SKILLS AND PROGRESS TO GOOD SKILLS.

Since FYW is a General Education program with a broad focus across disciplines, we have no specific disciplinary concepts for students to apply, per se. However, if we look from the perspective of some generalizable writing applications, then this seems an accurate assessment (OK).

We have little-no evidence of application of discipline concepts for AGS students, as most of their courses these students take are Gen Ed. Most do not have a concentration in a major.

Teaching plans are very difficult for students to complete because of their inability to understand and apply professional writing to their practice.

Again--since our program involves development across the undergraduate careers of the students, assessment would vary by student level. Early evidence from our new assessment device indicates, however, that we may do well to focus on general reading comprehension in particular--even for our honors students.

Good:

Independent study projects (e.g., BIO 598)

Faculty put students somewhere between okay and good on this item. Our theory course, research methods course, and senior thesis suggest that there is room for improvement in applying discipline-specific concepts in their own research.

Semester academic progress reports from participants' instructors.

Very Good:

Successful completion of a thesis requires that students be able to use discipline-specific concepts to first generate a novel research question (and write a research proposal) and then to use those concepts to explain the results of their research, and to explain what their study has added to what was known previously, in the discussion section of their thesis. The success our M.S. students have had in getting the results of their research published in peer-reviewed journals is one importance indication of how well our students have used the primary literature, and applied discipline-specific concepts, in their writing. Since 2008, our faculty and graduate students have published more than 80 papers in peer-reviewed journals. Perhaps the ultimate test of the ability of our M.S. students to apply discipline-specific concepts comes after graduation when they have biology-related jobs. Evidence of the ability of our M. S. graduates to apply what they have learning in our program includes the fact that, since 2008, 10 of our M. S. graduates have been hired as wildlife or aquatic biologists by state fish and wildlife agencies in Kentucky, Florida, and Washington state; two have been employed since 2008, three since 2009, two since 2010, and three since 2011. Other graduates in other positions have retained their positions since 2008 (e.g., biologist for the National Park Service), 2009 (e.g., environmental consulting firms), 2010 (e.g., community college instructors), and 2011 (e.g., director of Kentucky Heartwood). Although such evidence is only indirect, the continuing employment of many of our M. S. graduates suggests that employers are satisfied with their work and their ability to apply discipline-specific concepts in performing their duties.

ADDITIONAL COMMENTS

Most professional programs require a standardized exam for admission. All of these exams require a significant amount of reading of written text, and analysis and interpretation of scenarios presented in written form. Students need to be able to read and comprehend the test materials before they can answer the questions. They usually know the answers when quizzed verbally, but cannot comprehend the written questions. The MCAT was significantly revised this year. The MCAT developers inform us that reading comprehension is a big part of the new test. It is actually step #1--if students cannot comprehend the narratives presented on the test, they cannot hope to respond correctly. I have less experience with the other standardized exams required for health professions, but I suspect reading comprehension is vitally important on every exam.

Students take the LSAT before applying to law school. I see all EKU students' scores for that exam. The biggest weakness is reading comprehension. The LSAT and law school (and the professional of law) requires significant reading, analyzing, and interpreting of cases. If students do not have excellent critical reading skills they cannot do well on the LSAT, do not get into law school, and if they were to get it, they would not survive the law-school curriculum nor would they be a good lawyer. Students are able to apply discipline concepts fairly well, from my perspective. If you ask them, verbally, to apply concepts, they can often do a great job. However, they cannot critically read text in order to obtain sufficient information that is needed to apply concepts. Often, they don't understand the question because they have not interpreted the written text appropriately. The problem seems to be both a lack of general reading skills, e.g., they don't know the strategies for critical reading, and a lack of reading skills that are specific to various disciplines. They don't read a history text differently from a novel, for example. If we were to teach general reading skills in ENG 101/102 and focus on discipline-specific reading skills in every major, our students would be much better prepared for the LSAT and most other professional exams.

If we discuss concepts and use powerpoint, and non-reading/writing teaching methods, students tend to be able to demonstrate and discuss concepts; however, their ability to do so without prior oral discussion is concerning. 95% job placement in the field speaks for how well students are understanding EHS concepts.

Our point is that research exists to point toward a growing literacy problem. If kids don’t learn comprehensive reading strategies, how do we shape/teach to/mentor a mind devoid of information and shape it into knowledge? Admittedly, these strategies should have been employed in K-12, but they aren’t.

Some librarians have noted that information literacy is a higher-order thinking skill (even the skill of locating information, not to mention the still more advanced skill of evaluation and use of that information), that is very much dependent on one's ability to critically read written information.

PLEASE NOTE PREVIOUS COMMENTS AS THE HISTORY FACULTY IS DIVIDED IN ITS RATINGS. MOREOVER, ALTHOUGH HAVE OBSERVATIONAL EVIDENCE FOR THEIR RATINGS THEY EMPHASIZED THE WRITTEN WORK WE USE FOR PROGRAM EVALUATION.

Students have trouble picking out what is important in a reading. They read with surface-level attention, missing deeper information.

Nothing other than the obvious -- there is a great deal of individual variation across students.

Of the three potential QEP topics, reading comprehension most significantly affects students in this degree program. Because this is an AA degree most of the courses are Gen Ed, and there is little opportunity to observe how students might apply a discipline's concepts. We view application of concepts as a topic that is better suited to intervention by faculty in each program, rather than as a University-wide endeavor. Additionally, although information literacy is certainly an important need for all students, for AA-level students (and probably ALL students) addressing poor reading skills must be step #1. Students must be able to comprehend what they read before they can evaluate the information.

Adult students in this degree often return to school after years in the work force. Many of these students are relatively successful in their jobs, but are seeking to move up. They are surprised when they begin to struggle in their classes, and their difficulty often centers around poor reading skills. All of the courses in this program are offered online, and if students do not read well it negatively affects all of their coursework. Faculty report that students miss key points in lessons, misinterpret reading assignments, and complete papers incorrectly, often because they have not accurately comprehended the necessary text or instructions. Students report that they need help learning to read more effectively and efficiently.

Reading skills affect everything that students do. Students must be able to read to succeed in any class, major, or career. I think that EKU could make the biggest difference in students' success by focusing resources on reading skills from ENG 101, 102, and throughout the curriculum.

A centralized model of tutoring would help us address literacy skills and application skills across the curriculum. For example, if a student goes to a content-specific tutoring center that is reinforcing a disconnect between disciplines. Another example: A student who is a poor reader may not be able to make a connection between his/her poor reading skills and his/her inability to successfully complete a math word problem.

We (Kathy Williams & Lara Vance) feel that additional resources need to be funneled to students who lack critical reading, critical thinking and information literacy skills, including changes to classroom instruction (more supplemental education) and that these resources should not be limited solely to Eastern Bridge students, but perhaps be added to more first-year level courses in general. Lara suggests that university offices may want to review written communications they send out to students so that they are more "digestable," i.e. simpler to follow, keeping in mind that our students may have reading issues and we are communicating complex processes to them in a way that they may not be able to follow.