Carlo Magno, PhD Lasallian Institute for Development and Educational Research De La Salle...

Developing Data Driven Research in School Counseling

Carlo Magno, PhDLasallian Institute for Development and Educational ResearchDe La Salle University, Manila

Acitivity 1

Work in a group List down in the manila paper four

ways on how you use test/assessment data in your school.

Discussion during presentation

School Setting

EVIDENCES

Improvement of the teaching and learning process

Improvement of counseling processes

What do we need to develop among our students?

Learning and innovation skills

Creativity and InnovationCritical Thinking and Problem SolvingCommunication and Collaboration

Information, media, and technology skills

Information LiteracyMedia LiteracyICT (Information, Communications and Technology) Literacy

Life and career skillsFlexibility and AdaptabilityInitiative and Self-DirectionSocial and Cross-Cultural SkillsProductivity and AccountabilityLeadership and Responsibility

21st century skills

How do you get evidence that your students develop 21st century skills?

21st century

skills

AssessmentInternational Level Assessment

National Level

AssessmentRegional/District

Level

AssessmentClassroom Level

Role of counselors in School

Help generate assessment information about learners.

ASCA Standards

Counseling standards in the RP

What can be done with assessment data?

Informs development of academic programs and special programs

Informs further improvement of the curriculum Identifies sections/students that needs further

help Reflection on how to teach or deliver the

curriculum better Decisions on the allocation of resources and

priorities Informs what is happening in the schools

(academic standards)

What to report Achievement gains per class/section Achievement gains per subject area Achievement gains per level Trends: Comparison across school

years Trends: Comparison with other

schools/countries/states/region

Sources of Data in schools

Standardized tests Standards-based test Results from

inventories Teacher-made tests Interview data Data from teachers Data from parents

Scope of Presentation

Approach: Quantitative Design

Correlational Comparative Experimental

Correlational Studies

Involves two variables where one increases with the other

Examples: Grades and motivation: Does student motivation

increase with students’ grades? Attitude in Math and Math performance: Does

students’ attitude in math increase with their performance in math achievement test?

Math anxiety and test in math: Does anxiety decrease math test scores?

The choice between the variables should be guided by a theory (theoretical or conceptual framework).

Both variables should be quantitatively measured.

Correlational Studies

Linear Regression There is a straight line relationship

between variables X and Y When X increases, Y also increases-

positive relationship When X increases, Y decreases or vice

versa – negative relationship

Correlational Studies

Problem: Is there a significant relationship between achievement and aptitude?

Hypothesis: There is a significant relationship between achievement and aptitude

Relationship between achievement and aptitude

Achievement (X) Aptitude (Y)

100 99

95 98

90 94

85 87

82 84

80 81

75 78

70 73

65 68

50 60

Regression Line between achievement and aptitude

Scatterplot: X vs. Y

Y = 14.379 + .85633 * XCorrelation: r = .98966

40 50 60 70 80 90 100 110

X

55

60

65

70

75

80

85

90

95

100

105

Y

95% confidence

Laziness Perseverance

100 35

95 40

90 45

85 50

75 55

70 60

65 64

60 70

55 76

50 80

Relationship between laziness and perspeverance

Relationship between Laziness and Perseverance

Scatterplot: Y vs. X

X = 139.94 - 1.138 * YCorrelation: r = -.9959

30 40 50 60 70 80 90

Y

40

50

60

70

80

90

100

110

X

95% confidence

Correlational Studies Analysis

2 variables that are interval or ratio: Pearson r 2 variables are ordinal: Spearman rho 2 variables and each is a dichotomy: phi coefficient

High Satisfaction in teaching

Low satisfaction in teaching

High teaching performance

50 21

Low teaching performance

12 48

• A significant relationship occurs if scores are extreme enough to surpass the probability of error.•If p value is < obtained value: reject the null hypothesis•If the obtained value > critical value : reject the null hypothesis

Interpreting the correlation coefficient

Direction or direction Strength Significance Variance

Group Comparison Studies

Involves group formed in categories (2 or more) and these categories are compared on an characteristic.

The groups are called as the independent variable The characteristics of where the groups are compared

on are called as the dependent variable. Examples:

Is there a significant difference between males and females on their math performance?

Is there a significant difference between public and private school students in their study habits?

Are there a significant differences among the school ability of students from across three years (2010, 2011, 2012)?

Are there significant differences among teachers, administrators, and staff on their attitude towards the RH bill?

Group Comparison Studies

Take note that the IV... is categorical can have two or more levels can also be more than one.... Example: Can gender and socio-

economic status differentiate students general intelligence?

A theoretical or conceptual framework is needed to justify the comparison.

Group Comparison Studies

Case: Third year high school males and females are tested in their Mathematical Ability

Males Females26 3824 2618 2417 2418 3020 2218

Group Comparison Studies

Males: Mean = 20.14 SD=3.48

Females: Mean = 27.33 SD = 5.89

Mean of Males and females in Math

Box & Whisker Plot: Var2

Mean ±SD ±1.96*SD Males Females

Var1

12

14

16

18

20

22

24

26

28

30

32

34

36

38

40

Var2

Group Comparison Studies

1. H1= There is a significant difference between males and females on their math scores

2. =.05 df = N1 + N2 –2 df = 7 + 6 –2 df = 11 t critical value = 2.201

Group Comparison Studies

3. Computation

t = X1 - X2

x12 + x2

2 1 + 1 N1 + N2 – 2 N1 N2

t = - 2.73

Group Comparison Studies

4. Decision and Interpretation

Since the t obtained which is – 2.73 is greater than the t-critical which is 2.201, the null hypothesis is rejected.

This means that there is a significant difference between males and females in their math scores.

Females (M=27.33) significantly scored higher on math as compared to the males (M=20.14)

Group Comparison Studies

4. Decision and Interpretation (another way using p values)

Since the p value obtained which is 0.0195 is less than the alpha level which is .05, the null hypothesis is rejected.

This means that there is a significant difference between males and females on their math scores.

Females (M=27.33) significantly scored higher in math as compared to the males (M=20.14)

Factorial Design

Independent Variable

B

  A1 A2 A3   

B1 A1 B1 A2 B1 A3 B1 B1 Mean

Main Effect for B

B2 A1 B2 A2 B2 A3 B2 B2 Mean

  A1 Mean A2 Mean

A3 mean

  Main Effect for A  

Main effect of A

Main Effect of B

Interaction effect of A and B (A X B)

Talent

Achievement

Effect of Achievement and Type of school on Talent

Low Achievers

High Achievers

Type of school

Public school

Private School

H1: Achievement have a significant main effect on

talent (there is a significant difference between high

and low achievers on talent) Type of school have a significant main effect on

talent (there is a significant difference between public

and private school students in their talent) There is a significant interaction effect between

achievement and type of school (there are significant differences among high

achievers in public, high achievers in private, low achievers in public, and low achievers in private in their talent

Effect of Achievement and Type of school on Talent

Group Comparison Studies Analysis

If two categories are compared on one DV: t-test for two independent samples

If three or more categories (one IV) are compared on one DV: One way Analysis of Variance (ANOVA)

If two IV are investigated on one DV: two way ANOVA

If two or more IV are investigated on two or more DV: Multivariate Analysis of Variance (MANOVA)

Experimental Designs

Carlo Magno, PhDDe La Salle University, Manila

Effectiveness of an intervention on a set of measure (Experimental Study)

The effect of a treatment is tested on a specific change on a characteristic.

The treatment that is given to participants are called as the independent variable.

The independent variable should be manipulated. Ex. Groups are randomly assigned to listening and

watching stimulus to enhance their memory. Ex. Groups are randomly assigned to reading a text or

watching a news to enhance their recall of the information.

The characteristic that changes dues to the variation or manipulation of the IV is called as the dependent variable.

Experimental Study

How is the IV manipulated? Presence of absence Amount Type

Presence vs. absence

The effect of think-aloud reading on the reading comprehension of grade 8 students.

1st group: think-aloud while reading 2nd group: silent reading

Amount manipulation

The effect of cognitive load of concept on the recall of college students.

1st group: 200 words to study 2nd group: 500 words to study 3rd group: 800 words to study 4th group: 1,000 words to study 5th group: 1,200 words to study

Type manipulation

The effect of labeling on the teachers conduct assessment of students

ResultsTrouble makers low conductAverage Average conductIdeal students High conduct

Experimental Study

In an experiment done by dela Cruz, Cagandahan and Arciaga (2004), the effect of nonbehavioral intervention techniques was investigated on the computational abilities of fourth year high school students. The non-behavioral intervention techniques has three levels, bibliotherapy, small group interaction and games. These techniques were used as a teaching strategy in a lesson in a math class for three sections. Each of the strategy was used for each section. One section did not receive any strategy which served as the control group. After undergoing the strategy, the students were tested where they answered a series of computation items.

Experimental StudyBibliotherapy Small group

interactionGames Control Group

X1 X2 X3 X4

X1 X2 X3 X4

X1 X2 X3 X4

X1 X2 X3 X4

Experimental Study

1. H1: The non-behavioral intervention techniques have a significant effect on computational abilityH1: There are significant differences among the groups receiving bibliotherapy, small group interaction, games and control in their computational ability.

2. 2=.05 df between = groups – 1 = (4-1=3) df within = (N – 1) – df between ((209-1)-

3)=205 df total = df between + df within (3 + 205) F ratio critical value = 2.65

ANOVA Hypothesis Testing3. Computation

F ratio computed = 4.62

4. Decision and InterpretationSince the F ratio obtained which is 4.62 is

greater than the F ratio critical which is 2.65, the null hypothesis is rejected. The non-behavioral intervention techniques have a significant effect on computational ability.

ANOVA Hypothesis TestingIntervention techniques; LS Means

Current effect: F(3, 205)=4.6819, p=.00347Effective hypothesis decomposition

Vertical bars denote 0.95 confidence intervals

controlGames

BibliotherapySmall group interaction

Intervention techniques

3.5

4.0

4.5

5.0

5.5

6.0

6.5

7.0

7.5

8.0

8.5

com

puta

tion

The group who received the small group interaction significantly scored the highest among other intervention techniques.

Hypothesizing

The hypothesis needs to be backed up by a theory.

The effect of IV on the DV should be strictly controlled, no other factors should affect the DV other than the IV.

Extraeneous variables (show examples)

Techniques of constancy

Experimental Designs

Research Design – Refers to the outline, plan or strategy specifying the procedure to be used in seeking an answer to the research question

True Research Designs - Answers the research questions or adequately tests hypothesis. Extraneous variables are controlled Inclusion of a control group External validity - Generalizability

Experimental Designs

1. After-Only Design Dependent variable is measured only once and this

measurement occurs after the experimental conditions have been administered to the experimental group.  Treatment Response Measure

 Experimental Condition X YControl Condition Y

Between Subjects Design – If different subjects are used in each experimental treatment condition.

Within Subjects Design – If the same subjects are used in each experimental condition.

Experimental Designs 1.1 Between-Subjects After Only

Design subjects are randomly assigned to

the experimental and control group.

Simple Randomized Subjects Design Includes more than one level of the

independent variable

Experimental Designs

Factorial Design Two or more independent variables

are simultaneously studied to determine their independent and interactive effects on the dependent variables.

Main effect – influence of one independent variable

Interaction effect – Influence that one independent has on another

Experimental Designs

Within Subject After-Only Design Same subjects are repeatedly

assessed on the dependent variable after participating in all experimental treatment conditions

Experimental Designs

Combined Between- and Within-Subjects Designs

Factorial Design Based on a mixed Model Two independent variables have to be varied

in two different ways. One independent variable requires a

different group of subjects for each level of variation.

The other independent variable is constructed in such a way that all subjects have to take each level of variation.

Experimental Designs

Experimental Designs

2. Before-After Design The treatment effect is assessed by

comparing the difference between the experimental and control groups’ pre- and posttest scores.

The Solomon Four-Group Design - Designed to deal with a potential testing threat. - Testing threat occurs when the act of taking a test

affects how people score on a retest or posttest. - The design has four groups - Two of the groups receive the treatment and two

does not. - Two of the groups receive a pretest and two does

not. - By explicitly including testing as a factor in the

design, we are able to assess experimentally whether a testing threat is operating.

Experimental Designs

Switching Replications Design

- There is a need to deny the program to some participants through random assignment.

- A two group design with three waves of measurement. - The implementation of the treatment is repeated or

replicated. - In the repetition of the treatment, the two groups

switch roles: - The original control group becomes the treatment

group in phase 2 while the original treatment acts as the control. By the end of the study all participants have received

the treatment.

Experimental Designs

Randomized Block Design

Constructed to reduce noise or variance in the data Requires that the researcher to divide the sample

into relatively homogeneous subgroups or blocks. Then, the experimental design desired is

implemented within each block or homogeneous subgroup.

The key idea is that the variability within each block is less than the variability of the entire sample. Thus each estimate of the treatment effect within a block is more efficient than estimates across the entire sample

Experimental Designs

Recap

What are the three approaches in conducting a study?

Activity

Construct a plan for your classroom research Research Question Hypothesis What conceptual/theoretical framework will

be used? (be ready to explain) Method

▪ Experimental Design▪ Participants (who and how many)▪ Instruments used (how will you measure the DV?)▪ Procedure