Validity and Empirical Data

8/8/2019 Validity and Empirical Data

1/12


2/12

Founder

Sanford G. Kulkin, Ph.D

This program was researched,written and developed by Dr.Sanford Kulkin, Ph.D. andColeen Kulkin, M.A.

After 15 years of encouragingbehavioral strengths in corporations, managementconsultant and IML founder, Dr. Sanford G. Kulkinturned his sights toward educating young people in theprinciples of strength recognition.

Dr. Kulkin heads the development team forCAREERQuest Acclimation & Retentionmaterials. He is the Founder and President of TheInstitute for Motivational Living Inc. (IML), a training

and publishing company that designs and publishestools to help people communicate and work togethermore effectively. He is also the president ofCAREERQuest, the marketing company for IML.

Dr. Kulkin is a nationally recognized expert indeveloping individual ownership of corporate goals.He also serves as the Chairman of the Board ofDirectors for the Bair Foundation, a pioneering agencyin Christian foster childcare and is also Chairman ofthe Board of Family Life America, a non-profitorganization dedicated to enhancing the quality ofcontemporary family life. Dr. Kulkin earned hisBachelor of Arts degree from Washington andJefferson University and completed his doctorate inPsychology and Counseling at Vermont College ofNorwich University. Dr. Kulkin's goals are firmlyrooted in the success of every young man and woman.His vision is to help identify the strengths of everyindividual and direct him or her toward a successfulcareer.

CAREERQuest Acclimation & Retentions Materials are use in this study.

Coleen Kulkin is the Program Director for the

CAREERQuest retention and acclimation program. Sheis responsible for the research, and development ofprofiles, surveys, and content found within theCAREERQuest program materials. Ms. Kulkin hasdevoted much of her academic life to the study ofindividual learning, thinking, and communicatingpreferences.

Today, Ms. Kulkin serves as the Program Director forCAREERQuest and continues to be instrumental in theongoing development of retention and acclimation materi-als being used in many colleges and universities in theUnited States and Canada.

Ms. Kulkin earned her Bachelor's ofArts degree in Organizational Com-munication and her Master of Arts

degree in Corporate Communication,from Duquesne University. Duringher masters' studies, she participatedin the graduate assistantship program,assisting in the teaching of classes inbehavior modification and conflictresolution. Coleen is the daughter ofDr. Sanford Kulkin.

Co-AuthorColeen R. Kulkin, MA

The CAREERQuest program was researched, written and developedby Dr. Sanford Kulkin, Ph.D. and Coleen Kulkin, M.A.

2

Table 8

T-Test Comparison of Instructor Ratings Between Career Quest Participants and Non-

Participants

Participants Non-ParticipantsItem Mean Score Mean Score df t

The way the instructor 5.52 5.18 270 1.75 (N.S.)taught the course.

The total advising 5.68 4.65 270 4.45 ***experience with yourinstructor.

Your instructors 5.96 5.69 270 1.57 (N.S.)preparation for class.

Your instructors 6.03 5.46 270 2.95 **availability when youneeded help.

The helpfulness of your 5.64 5.17 270 2.23 *instructor in discussingyour personal success plan.

Your instructors concern 5.75 5.54 270 1.10 (N.S.)for your success.

Your instructors 5.99 5.48 270 2.81 **enthusiasm for the course.

The overall quality of 5.70 4.54 270 5.99 ***class discussions andactivities.*** p < .001** p < .01* p < .05N.S. = not significant

23


3/12

Table 7 continued

optimal learningenvironment for myselfbased on the way Iprocess information.

I can identify charact eristics 5.21 4.85 270 1.99 *of my style of thinking.

I understand why 5.52 5.23 270 1.62 (N.S.)

different personality

styles can come intoconflict.

I know my learning style. 5.62 4.94 270 3.51 ***

I know how to adapt to 4.69 4.32 270 2.02 *classes that are not idealfor my thinking style.

I am aware of my strengths 5.35 5.28 270 .41 (N.S.)and weaknesses in relatingto others.*** p < .001** p < .01

* p < .05N.S. = not significant

22

Index

Introduction Pg. 3-5

Testimonials Pg. 6-7

Eastern College Validity Study Pg. 8-23

3


4/12

Introduction

CAREERQuest was established in 1996 to market and distribute career-related products and servicespublished by The Institute for Motivational Living Inc. CAREERQuest is devoted to building students'self-esteem through outreach and college programs like the retention and acclimation program that thisempirical study addresses. CAREERQuest was founded to help today's young men and women reachtheir fullest potential.

From its inception, CAREERQuest has been testing retention and acclimation in hundreds of highschools, colleges and universities in the United States and Canada. For the past three years,CAREERQuest materials have been enhancing the socialization and learning processes of thousands ofstudents while positively impacting retention rates. In a market that is dominated by costly and complex

retention programs, CAREERQuest has been seen as a breakthrough solution. CAREERQuest hasbeen able to deliver products that are cost-effective, user-friendly, and simple to administer and interpret.Most importantly, the CAREERQuest program is easy to understand because it has been written foryoung people.Today, new electronic methods for delivering CAREERQuest materials are available allowing colleges,universities and young professionals access to these products via the Internet or college Intranet.CAREERQuest will continue to market and distribute high-quality products in the educational,professional, church, and business arena.

In 1997 the Wall Street Journal published the results of an ACT (American College Testing) study showinga record high college drop out rate of 27%. This was especially alarming, since colleges and universitieshave made retention and acclimation a major focus before, during and after orientation.The CAREERQuest Program was developed in 1996 by Dr. Kulkin to address these very issues thatcolleges across the country were facing.

In an effort to battle decreasing retention rates, the CAREERQuest program identifies unique factors in

each student, allowing individual communication and learning needs to be met by advisors and faculty. Theprogram targets individual learning modalities by assessing personality, learning, thinking, and valuesstyles as well as targeting career and goal choices. The key to CAREERQuest success is in giving stu-dents "individualized" attention, dealing one on one with students who believe "they have no sense ofbelonging," "can't handle the academics," or "are unsure about their goals and career choice."

Beginning in 1997 CAREERQuest implemented its innovative workbook based assessment system. Ithas been adopted by numerous institutions including both secular and religious colleges, universities, highschools, home schools, and several specialized programs. Many colleges and universities are usingCAREERQuest to enhance their freshmen orientation program as well as aiding their senior class andgraduate students to better understand what today's employers are looking for in individuals. As a "College101" course, CAREERQuest is a springboard for understanding oneself. Once students know their ownstrengths and preferences, they will be better equipped to handle transitions, understand diversity, and makecritical decisions regarding their future. Advisors also find the CAREERQuest program helpful forlaunching discussions about academic direction. Helping incoming students adapt to college life has alsoaided colleges in their efforts to combat decreasing retention rates. For the past three years,CAREERQuest has made a tremendous difference in the academic and socialization process of thousandsof students.

4

Table 7

T-Test Comparison of Student Outcomes Between Career Quest Participants and Non-

Participants

Participants Non-ParticipantsItem Mean Score Mean Score df tI have learned to set 4.82 4.37 269 2.21 *realistic goals thissemester as a resultof this class.

I understand my style 5.26 4.73 269 2.76**of taking in andprocessing information.

I recognize how to study 4.97 4.82 270 .83 (N.S.)in a way most helpfulto my learning style.

I have a better under- 5.16 4.98 270 .99 (N.S.)standing of people whoare different from me.

I can identify my 5.59 5.17 270 2.19 *personality type or style.

I can identify the 5.43 5.14 270 1.60 (N.S.)environment in which Ilearn best.

I can recognize personality 5.13 5.07 270 .32 (N.S.)types or styles differentfrom my own.

I know how to adapt to 4.69 4.34 270 1.94 (N.S.)professors whose teachingstyle differs from mylearning style.

I understand my strengths 5.26 5.32 270 -.37 (N.S.)and challenges as alearner.

I can identify others 4.60 4.65 270 -.29 (N.S.)style of thinking.

I know how to create an 4.64 4.69 270 -.25 (N.S.)

21


5/12

Table 6

T-Test Comparison of Career Items Between Career Quest Participants

and Non-Participants

Participants Non-ParticipantsItem Mean Score Mean Score df t

In thinking about a career, 5.34 5.27 270 .36 (N.S.)I have examined mystrengths and values.

I know several career 5.24 4.92 270 1.57 (N.S.)possibilities that may beoptimal for me.

In the process of 5.53 5.32 270 1.18 (N.S.)thinking about whatcareer might be a goodoption for me, I havethought about whatthings are most importantto me.

I feel confident when I 4.91 4.68 270 1.05 (N.S.)think about decidingon a career and a major.

Note: N.S. indicates non-significant result

20

The CAREERQuest program was developed from over 15 years of experience in the developmuse of psychometric instruments. The last five years have been spent applying this experience to thdevelopment and testing of our college retention and acclimation program. Our experience in thof behavior, personality, learning, and goal setting makes us the forerunner in our field.

The 2 year study of CAREERQuest materials has just been completed by Eastern Collegproven through The Council for Christian Colleges and Universities (CCCU) "Affirming Strengtstudy that retention was increased by 7%. Today this program is a premiere source for college rand acclimation support.

These materials have been written to meet the needs and proficiency levels of high school sencollege freshmen. CAREERQuest assists schools by keeping students on campus until grad

increasing both enrollment and profitability at colleges and universities. As a resCAREERQuest program may also help combat student dropout rates. Our program aids a stunderstanding themes relating to "the self". Students who are familiar with their natural prefemotivations, values, ideal environments, greatest fears, and learning/thinking strengths as well aable to identify those of others, will be better equipped to adjust to college life. Students also gaininto reducing the chance of conflict with roommates and increasing understanding of teachteaching techniques. CQ also helps in Diversity training (appreciation for differences in others). CAREERQuest addresses the positive and negative "dif ferences" in all people; it has thelement of educating students on the importance of "recognizing" and "appreciating" that whicpeople unique and valuable.

Leadership opportunity for student mentors

For many schools, student volunteer mentors or stipend-paid mentors are finding implementingCAREERQuest to be a leadership challenge. Many senior mentors are creatively administerprogram with freshmen, providing guidance, and making new friendships as they bond with i

students. Mentors are finding this experience to be both rewarding and a leadership achievement their resume.

Peace of mind for parents or financiers

The program adds value to the higher education investment. In particular, parents who financhildrens education will see the institution as being pro-active and accountable because thprovided specialized career development training. Time and money are also saved with fewer chthe student's curriculum direction.

Encouragement of study habits

From the start (I.e. freshman orientation) its a good idea to practice effective study, writtest-taking habits. CAREERQuest attempts to target each students preferences toward learnioffer study tips and practices that will help students maximize their learning strengths.

Aids students decision-making capacity

A student will make some of the most critical and consequential decisions of her life during hecollege. This is quite a task for most young people as they advance to adulthood. For some, just gclass on time is a struggle during the first year on campus. If the writing is not "on the wall,"

5


6/12

Career DISCovery, through assessments, reveals an individuals natural preferences so that he/she can makeinformed and realistic academic and career decisions.

Enhances student-teacher interactions

Because CAREERQuest materials offer practical information regarding students' communication andlearning strengths; students are able to gain credibility with professors and develop learning tactics, makingthem more effective in the classroom.

Enhances student-advisor interactions

Some colleges and universities are using CAREERQuest materials as a springboard for academic

planning discussions with advisors. Assessments, surveys, and reports in CAREERQuest paint a clearand accurate picture of an individual's strengths, limitations, preferences, choices, and goals. Advisors whoapply this information to a realistic academic path will stand a better chance of gaining commitment fromthe student.

Enhances student-student interactions

The transition from high school to college is a major shift for most young people. Consider that their web ofhuman interaction is blossoming. Students meet new diverse people from disparate backgrounds andcultures in class while roommates are often chosen randomly. Because college life can seem lonely andintimidating, information in CAREERQuest provides guidance in understanding the personalities,motivations, and values of peers.

Additional Benefits

The CAREERQuest program is easy to use, understand, administer, and interpret results.

Practical program

The profiles, graphs, surveys, and reports generated from our program have practical applications in the realworld. Advisors, parents, faculty and students can begin to apply this information and enhance relatio n-ships, study skills, or make realistic and effective career decisions immediately. Reports are designed to bepractical and applicable to the freshmen first year experience.

Facilitator-teacher curriculum available

Leader's Guides (paper): Leader's guides give teachers the curriculum to facilitate a one to three creditcollege course.CD-ROM: "Train-the-trainer" training is available as an interactive CD-ROM.For more information see Training.

Various media options available

The Career DISCovery program is available in paper, on the Internet, via a corporate Intranet, or with aCD-ROM.

6

Table 5

T-Test Comparisons of the SACQ Scale Scores of Career Quest Participants versus Non-

Participants

CQ Participants Non-Participant sSACQ Scale Mean Score Mean Score df tAcademic 145.4 143.7 228 .44 (N.S.)Adjustment

Attachment 107.7 107.0 228 .25 (N.S.)

Personal/Emot ional 91.5 89.8 228 .62 (N.S.)Adjustment

Social Adjustment 137.3 134.4 228 .81 (N.S.)

Full Scale 431.1 430.3 228 .08(N.S.)Note: N.S. indicates non-significant result

19


7/12

Item Mean Rating % Rating 1 or 2 % Rating 6 or 7

I have learned some information about the 21st

century workforce requirements . 5.52 .7% 56.2%

I am aware of careers or jobs that match my

personality style. 5.80* 1.4% 67.9%

I understand my general career preferences . 5.69 2.8% 63.9%

I can consider carefully what careers will best suit my style. 5.66 1.4% 65.3%

I have identified skills and abilities for my career preferences. 5.54* 1.4% 59%

The workbook was easy to use. 6.10 1.4% 77.7%The material was easy to understand. 6.21 1.4% 81.1%

The content was interestin g. 5.66 4.2% 60.2%

I enjoyed the material. 5.47 4.9% 56.7%

I liked learning about myself. 5.73 2.8% 66.2%

The professor made the class interestin g. 6.09 0 75.6%

The professor was enthusiastic about the material. 6.09 .7% 75.6%

*= Statistically significant difference in student responses depending on the instructor (p .05)

Table 4

Student Reponses to the Career Choice Assessment Session

18

Testimonials

In the past, CAREERQuest has received feedback from students, administrators, and fFollowing are comments from faculty, parents, and students who have experienced our reteacclimation materials.Career DISCovery, through assessments, reveals an individuals natural preferences so that he/sheinformed and realistic academic and career decisions.

From Educators

"I taught one section of Intro to College. This term, I had 41 students and we used The CognitivStyle Workbook, The Perceptual Learning Style Workbook, and The Goal Setting Workbook. I

teaching this course for six y ears and never before have my students responded more positimaterials we have used."-Ed Casteldine, Freshmen Director, Northwest Nazarene College, Napa, Idaho

"We have used the CAREERQuest materials for one semester only. It is new to us so we are way around the best way of presenting it to our students. We offered the materials in seminar formThe materials themselves are very easy for the student to comprehend, and they enjoy the discovethemselves that each workbook fosters. The materials were also very adaptable and could eascourses already being ta ught or new ones being developed. It could very easily be used outsitime as well, in a seminar or workshop format.We are hoping that over the course of time this material will help acclimate our students to our well as help them feel a stronger sense of direction in their lives because of the considerationgiven to the different processes."-Lisa Frankland, Director of Freshmen Orientation, Briercrest Bible College, Canada

"We have been using CAREERQuest materials for three years, and we are beginning to s

results in acclimating and retaining students at Geneva. The materials have helped our studareas of life: social, academic, and personal development. I believe these materials need to bschool in America. Our Freshman Program has helped retain our student body and saved usmoney."-Joy Jewell, Vice President of Student Development, Geneva College, Pennsylvania

"We utilized the material during our Freshman Weekend Retreat. After we went through the PWorkbook, the students began to relate what style they were with each other. It was the "buzzthe entire campus after the retreat!"

-Gary Ewing, Freshman Orientation Director, Colorado Christian University, Colorado

"These materials are tremendous! They have helped our students understand thepersonality and learning styles CAREERQuest is one of my strongest retention tools."

-Joel Wilson, Freshman Director, The Criswell College, Texas

From Parents

My evaluation of the CAREERQuest program was positive . I felt the students learneinformation about their learning styles with the workbooks."-Barb Streeter, Director of Freshman Programs, Gettysburg College, Pennsylvania

7


8/12

"When my child brought the materials home during the break, we opened new areas ofcommunication and saw the differences within our family."

-M.M., Pennsylvania"This program adds value to the cost of my child's education."

-K.L., British Columbia

From Students

"This past year, I trained mentors in this material for Genevas freshmen orientation program and noticedthat both mentors and freshmen had gotten excited about learning the CAREERQuest material. Whenthey went through the workbooks, you could see the light bulbs being turned on in their minds. Thementors really had a lot of fun teaching CAREERQuest to the incoming students and it rubbed off

on the freshmen. One very outgoing freshman introduced himself at lunch by saying, "Hi! Im Michael,and Im a high I!" In my focus group, so many new relationships have developed. Weve openly talkedabout differences, conflict, studying, dating, roommates and family. I have become good friends with someof the people whom I mentored and I know these materials built the relationships foundation."-Debra Csellar, Senior Mentor and Student, Geneva College, Pennsylvania

"Thanks for coming to class and actually telling us about ourselves . I really understand what type ofperson I am and even what type of person I want to be . It seems now that every time I argue, laugh ortalk with someone that I can really tell what type of a person Im dealing with. It also helps me have acloser; more open relationship with people and talk to them more comfortably. I really had fun learningabout personality types and hope to develop more of an understanding the older I get."-Audrey Denzer (written to Duane Storey, CAREERQuest consultant), Senior at Sis ters High School,Sisters, Oregon

"I finally realized the differences in myself and my friends. In fact, I just used this information with a

relationship this past weekend."

-S.C., Oregon

"My roommate and I have many differences and there had been virtually no communication with eachother. However, once we went through the materials, our communication levels have increased andwe understand our differences."

-P.T., Pennsylvania

"I had a difficult time getting acclimated from the high school to a new learning environment. With thelearning workbooks, I have identified my learning and processing styles and have seen my grades

improve dramatically."

-T.T., West Virginia

8

Item Mean Rating % Rating 1 or 2 % Rating 6 or

I understand the general characteristics of my

thinking style. 5.91 .7% 72.8%

I recognize that there are strengths and

limitations of my own thinking style. 6.03 .7% 77.0%

I have gained new insights about myself. 5.31* 5.4% 53.1%

I have identified a way to improve how I 5.09 6.1% 43.9%

I can identify the thinking styles of others. 5.07 2.7% 39.9%

The workbook was easy to use. 6.18 1.4% 81.6%

The material was easy to understa nd. 6.17 1.4% 82.3%

The content was interesti ng. 5.46 3.4% 57.2%

I enjoyed the material. 5.40* 4.1% 53.1%


The professor made the class interestin g. 6.03 0 76.2%

The professor was enthusiastic about the material. 6.02* 0 74.9%

*= Statistically significant difference in student response depending on the instructor (p .05)

Table 3

Student Responses to the Cognitive Thinking Style Assessment Session

17


9/12

Item Mean Rating % Rating 1 or 2 % Rating 6 or 7

I have identified my own learning style. 5.79 4.2% 66%

I have learned some strategies to enhance learning. 5.44 5.4% 53.3%

I am aware that my professors teaching styles may

not match my learning style. 6.07 0 79.7%

I can identify the learning styles of others. 5.04 5.4% 38.1%

I can create an ideal learning environment. 5.43 .7% 54.1%

The workbook was easy to use. 6.22 .7% 81.1%

The material was easy to understand. 6.27 0 84.4%

The content was interesti ng. 5.34 6.8% 49.3%

I enjoyed the material. 5.26 7.4% 50.0%


The professor made the class interestin g. 6.15* .7% 83.8%

The professor was enthusiastic about the material. 6.01* .7% 76.4%


Table 2

Student Response to the Perceptual Learning Style Assessment Session

16

EASTERN COLLEGE

A STUDY OF THE VALIDITY

OF

CAREERQuests Freshman Acclimation and Retention

Program and its Impact on First-Year Student Success

& Retention

Laurie A. Schreiner, Ph.D

Professor and Chair of Psychology

9


10/12

EMPIRICAL STUDY SUMMARY

The Career Quest Program

The Career Quest Program (CQ) is a self-administering and self-interpreting program designed toenable students to become better acclimated to college and prospective careers by understandingthemselves more thoroughly.

The program, published by The Career Quest Corporation, contains five sections based upon theprinciples of personality styles, learning styles, thinking styles, career choices, and goal setting.Four of the five sections contain an assessment instrument that determines an individual's unique"style." Individual styles are then interpreted and students are given action plans and practical

advice for improving habits, behaviors, and making informed decisions based upon strengths andlimitations.

Purpose of the study

The purpose of the study was to determine if using Career Quest's Freshman Acclimation andRetention Program actually resulted in significant differences in student retention rates, academicperformance, and readiness for career selection, and acclimation to college.

The CQ Program is part of a longitudinal study conducted on behalf of the Coalition of ChristianColleges and Universities (CCCU).

Method of the study

Eastern College, a Christian liberal arts college located in the suburbs of Philadelphia, was used as atest site for the CQ Program during the fall of 1998. Sixteen instructors of the first-year seminar

entitled Living and Learning in Community, a course focusing on a multi-faceted approach to wellness and spiritual wellness, were randomly assigned to one of two conditions. In the controlcondition, the instructors taught the three-credit course as it has been taught for the past four years,utilizing a common syllabus which did not include sessions on career issues, personality styles, orlearning styles. In the experimental condition, the instructors added four sessions of Career Questmaterials, but otherwise taught the course as the control condition instructors did.

Study participants

325 first-year students participated in full. Of them, 155 students participated in the experimentalcondition (exposure to CQ) and 170 students participated in the control condition (no exposure toCQ). Each signed an informed consent form indicating their agreement to participate in the researchstudy and to provide the feedback requested.

10

Feedback from students who participated in the follow-up interview in the Spring indicate

-50.7% found CQ to be helpful in more than one way-25.4% said CQ helped them a little-23.9% said CQ did not help them at all

Those who claimed little or no benefit from participating in CQ also claimed that they alrthis information about themselves before coming to college.

Item Mean Rating % rating 1 or 2 % rating

I have a better understanding of the strengths

and limitations of my personality style. 5.21* 4.0% 48.0%

I have gained new insights about myself. 4.93 5.4% 38.0%

I can identify the personalit y styles of others. 5.07 4.4% 42.7%

I am aware of how other peoples styles

affect me. 5.03 2.6% 36.7%

I have received insights on how to relate to

others who have different styles than my own. 5.00* 4.7% 38.6%

The workbook was easy to use. 6.26 .7% 82.7%

The material was easy to understand . 6.37 .7% 88.0%

The content was interestin g. 5.71* 4.2% 66.6%

I enjoyed the material. 5.49* 4.1% 58.7%


The professor made the class interesting . 6.19 0 82.6%

The professor was enthusiastic about the material. 6.21* 0 81.4%


Table 1

Student Responses to the Personality Assessment Session

15


11/12

Outcome tested: Participating in the CQ program leads to higher levels of student retentionMethod used: Chi-square analysisTo test the claim that CQ would have an impact on retention, the first-to-second-year retention rateswere compared to the rates of non-participants using a Chi-square analysis. Of the 155 students whoparticipated in Career Quest, 118 were retained after one year (76.1%); of the 170 students who did notparticipate, 111 were retained (65.3%), and that difference was significant (p


12/12

ment to the institution.

Course evaluation formsCourse evaluation forms were designed in consultation with the publisher of CQ to be ad-ministered at the end of each CQ class session. The form contained four items which hadbeen adapted from The Career Factors Inventory (Chartrand, Robbins, & Morrill, 1997), areliable and valid instrument designed to assess the level of students' readiness to choose acareer.

Procedure

Instructors who had been randomly assigned to include CQ in their course used four class sessionsin early September 1998 to introduce the materials and assist students in completing the inventoriesand interpreting their results. Instructors had received a full day of training by a CQ trainer in Au-gust 1998 and used instructor's guides to help them plan activities around the material. At the end ofeach CQ session, students completed a brief questionnaire asking them for feedback about their ex-perience with the material. Instructors who were not randomly assigned to the CQ condition contin-ued to teach the course the same way it had been taught for the past four years, using a common syl-labus and texts.

In October 1998, all firs t-year students participating in the study completed the SACQ question-naire. In addition, residence hall directors contacted each student individually if their SACQ indi-cated that they were experiencing difficulty adjusting to college, and a retention contact report formwas filed with the project director. At midterm, all students in all sections of the course created apersonal success plan that outlined their strengths and weaknesses and established goals for theirfirst year of college. At the end of the semester, a course evaluation was given to all students inboth conditions, to assess the outcomes of the course and to test for any significant differences in

outcomes for the CQ participant s. In the Spring of 1999, CQ instructors contacted all their studentsto conduct a follow-up interview on how students were using the information gained from CQ. Sev-enty-two students (46.5%) who had participated in CQ sessions came back for a follow-up interviewand provided feedback using a special form prepared for that purpose. In the Fall of 1999, the en-rollment of each student was obtained to ascertain the success of CQ as a retention aid.

Results

Reliability of CQ instruments

Test-retest reliabilit y is a pre-requisite for a valid testing instrument. To assess the reliability asconsistency over time, students' responses to the Personality Style Assessment were compared totheir responses on the Career Choice Assessment give two weeks later, since The Career ChoiceWorkbook uses the same behavioral dimensions as The Personality Style Workbook. Students' twohighest styles were recorded, and:

-49.3% of the students styles remained exactly the same over the two-week period-20.9% had the same top two styles, but in reverse order-23.1% had the same tip style but their second style had changed-8.3% had no agreement in their styles over the two-week period

12

Thus, 72.4% of the students' top personality styles remained constant over that period a20.9% changed their top style to become their second style. This indicated an acceptabreliability since chance alone would indicate that only 25% would maintain their top styleand only 6% would maintain both the top and second styles over time.

Validity of CQ instruments

Validity of the instruments is essentially a determination of whether or not it measurclaims to measure and its ability to accurately predict current of future behavior. There wmeasures that were used to assess the predictive validity of CQ instruments. CQ claims outcomes that students will experience as a result of participating in the program:

-improved study skills-better adjustment or acclimation to college life-higher confidence in their ability to make career choices-higher levels of student retention

Outcome tested: Participating in the CQ program leads to improved study skillsMethod used: Analysis of VarianceTo test this claim, the first-semester GPAs of students who participated in CQ were cothose students who did not participate. After controlling for high school GPA and SAT/Aupon college entrance, an analysis of variance indicated:

First-semester GPA scores showed significant difference-CQ participants had an average first-semester GPA of 2.85-Non-participants had an average first-semester GPA of 2.64-This shows a significant difference in first-semester GPA scores (F=3.934

First-year cumulative GPA scores showed less significant difference-CQ participants had an average first-year cumulative GPA of 2.77-Non-participants had an average first-year cumulative GPA of 2.55-This shows less significant difference in first-year cumulative GPA scores(F = 3.498; p = .08).

Outcome tested: Participating in the CQ program leads to better acclimation to college lifMethod used: t- testTo test this claim, the scores of CQ participants on the Student Adaptation to College Que(SACQ) were compared to the non-participants via a t-test. The SACQ was adapproximately two weeks after the last CQ session. There were no significant differenceCQ participants and non-participants on any of the scales of the SACQ or on the total scale

Outcome tested: Participating in the CQ program leads to higher confidence in studentmake career choices

Method used: t- testTo test this claim, four items were adapted from the Career Factors Inventory (Chartrand& Morrill, 997) and included in the final course evaluation given to all students. Aconducted to determine if there was a significant difference in the responses of students participated in CQ, compared to those who had not. There were no significant differenceCQ participants and non-participants.

13

Documents

Validity and Empirical Data