Contemporary Research Practices Carlo Magno, PhD.

2nd NATIONAL CONFERENCE ON COMPUTERIZED STATISTICAL MODELING AND RESEARCH

METHODS

Tagbilaran City, BOHOL May 6-8, 2015

5th NATIONAL SEMINAR-WORKSHOP ON ACTION RESEARCH AND

COMPUTERIZED ITEM/TEST ANALYSIS AND STATISTICS

Contemporary Trends on 10 Perspectives

1. Research is written for an international audience 2. Research is becoming an interdisciplinary process 3. Research designs are based on the needs of the

researchers 4. Identify sample size based on power analysis 5. Adapting the instrument necessary for the sample 6. Instruments validity and reliability computed using the

sample data 7. Computation of the effect size with the NHST 8. Thesis and dissertation beyond the IPO 9. Publishing research reports in international abstracted

and indexed journals 10. Concern on author impact factor

Research written for an international audience

– Initiatives supporting ASEAN Integration

– Research databases easily accessible in the internet

– Implication:

• Specifying whether events are emic (part of the social group) or etic (perspective of an observer)

• Is there a need to separate local and foreign reviews?

• Using more of published studies than books…

• Country of origin in describing the participants

Research becoming an interdisciplinary process

• Looking at the complementary across and within fields: – education and psychology

– biology and education

– education and computer science

• “Communities of practice” generating knowledge

• New shared practices are discovered

• Apprenticeships, participations, and worldviews are developed.

Research designs based on the needs of the researchers

• Nonexperimental designs according to Johnson (2001)

Identifying sample size based on power analysis

No real effect Real Effect

Reject H0 Type 1 error α (.01, .05)

Ho not rejected Type 2 error β (small as possible)

1-β Statistical

power

Slim chance of concluding that the treatment is

effective, despite the fact that it is

Statistical Power

• β=.20 (the error of rejecting a true Ho is 4x more serious than the error of not rejecting a false Ho)

• .80=acceptable power

Statistical Power

• The probability of rejecting a false null hypothesis.

• The likelihood that a study will detect an effect when there is an effect to be detected.

• If statistical power is high, the probability of making a Type II error, (or concluding there is no effect when, in fact, there is one) goes down.

Statistical Power

• The power of any test of statistical significance will be affected by four main parameters:

– the effect size

– the sample size (N)

– the alpha significance criterion (α)

– statistical power, or the chosen or implied beta (β)

Statistics Power small medium Large

r .80 26 63 393

t .80 29 85 781

Statistical Power

http://danielsoper.com/statcalc3/default.aspx http://www.statisticalsolutions.net/pss_calc.php https://www.dssresearch.com/KnowledgeCenter/toolkitcalculators/statisticalpowercalculators.aspx http://homepage.stat.uiowa.edu/~rlenth/Power/

Influence of Effect Size on Power

High school n=65 College n=153

Influence of Effect Size on Power

Influence of Effect Size on Power

N=82 Taiwanese in Taiwan N=98 Taiwanese in the Philippines

Influence of Effect Size on Power

• What inference can be gained between effect size and power with fixed sample size and alpha level?

Influence of Significance Level on Power

• Study of De Frias, Dixon, and Strauss (2006)

• N=418

• r=.14 (not significant)

• α=.01 (power=.23)

α=.05 Power=.45

α=.10 Power=.58

α=.15 Power=.65

α=.20 Power=.71

• What inference can be gained between level of significance and power with fixed sample size?

Influence of Sample size on Power

Magno (2005) Monitoring and metacognition

N=280 r=.14 Power=.65

Magno, Mamauag, & Parinas (2007) Independence and self-esteem

N=373 r=.14 Power=.78

Chemers, Hu, & Garcia (2001) Challenge-threat and self-efficacy

N=381 r=.15 Power=.83

Influence of Sample size on Power

Adapting the instruments necessary for the sample

• Producing instruments that measure target constructs adequately in target cultures

• What is the American equivalent of the Dutch item “Hoe heet de koningin van Nederland?” (Suppose that

item is part of a test of crystallized intelligence) • Literal/close translation: What is the name of the queen

of the Netherlands?” – Problem: Item more difficult for American children than for

Dutch children • Adaptation: “What is the name of the president of the

USA?” – Problem: Queen and president are not equally known in their

respective countries

Construct equivalence

• Similarity of construct in source and target culture

Cultural equivalence

• Norms about interaction (modes of address)

• “Cultural fact sheet”

Linguistic equivalence

• Translation accuracy: Retention of denotation and connotation

Measurement equivalence

• Retention of psychometric features (response styles)

• Similarity of factors measured by a test and comparability of scores

Four Important Perspectives (Harkness & Van de Vijver, 2012):

Instruments’ validity and reliability recomputed using one’s data

• It is customary that in every instrument used, evidence of validity and reliability are reported for the instrument used.

• Confirmatory Factor Analysis is conducted for multidimensional measures.

• Internal consistency reported – Cronbach’s alpha – Coefficient Theta – Coefficient Omega

• Accuracy of the instrument is supported by using IRT and polytomous IRT models for scales

Computation of effect size with the NHST

• Issues on NHST • 1. NHST does not provide the information which the

researcher wants to obtain • 2. Logical problems derived from the probabilistic

nature of NHST. • 3. NHST does not enable psychological theories to be

tested. • 4. The fallacy of replication. • 5. NHST fails to provide useful information because

H0 is always false. • 6. Problems associated with the dichotomous

decision to reject/not reject the H0. • 7. NHST impedes the advance of knowledge.

1X

2X

n df SD1 SD2 t

p

value

30 28 4.6 4.1 2.3 2.3 0.60 0.56

60 58 4.6 4.1 2.3 2.3 0.84 0.4

100 98 4.6 4.1 2.3 2.3 1.09 0.28

500 498 4.6 4.1 2.3 2.3 2.43 0.02*

1000 998 4.6 4.1 2.3 2.3 3.44 0.00*

A researcher wanted to look at the effect of behavior modification technique on the aggression of clients. A group of participants in the experimental group were given behavior modification technique and no treatment in the control. The aggression of the two groups were measured after.

Effect Size

• Cohen (1988) defines the effect size as the extent to which the phenomenon is found within the population or, in the context of statistical significance testing, the degree to which the H0 is false.

• Snyder and Lawson (1993) argue that the effect size indicates the extent to which the dependent variable can be controlled, predicted and explained by the independent variable(s).

Effect Size Measures • Effect size measures of Two In/dependent Groups

– Cohen’s d – Hedges g – Glass Delta

• Correlation Measure of Effect Size – r – χ2 ►Φ; t ► r; F ► r; d ► r

• Effect size for Analysis of Variance – Eta Squared – Omega Square Index of Strength – Intercalss correlation

22

21

2

21

ss

MMd

2

22

1

12

21

ns

ns

MMt

Cohen’s d Formula t-test for independent Means Formula

Cohen's Standard Effect Size Percentile Standing Percent of

Nonoverlap

2 97.7 81.10%

1.9 97.1 79.40%

1.8 96.4 77.40%

1.7 95.5 75.40%

1.6 94.5 73.10%

1.5 93.3 70.70%

1.4 91.9 68.10%

1.3 90 65.30%

1.2 88 62.20%

1.1 86 58.90%

1 84 55.40%

0.9 82 51.60%

LARGE 0.8 79 47.40%

0.7 76 43.00%

0.6 73 38.20%

MEDIUM 0.5 69 33.00%

0.4 66 27.40%

0.3 62 21.30%

SMALL 0.2 58 14.70%

0.1 54 7.70%

0 50 0%

Thesis and dissertation beyond weighted means, difference, and

relationships

• Bad practices

– All students uses the INPUT-PROCESS-OUTPUT in their theoretical framework

– Research questions #1 determined the demographic profile of participants (gender, SES, age, family structure, residence, etc)

– The factors of the instrument are physical, emotional, social, spiritual …

Publishing research in abstracted and indexed journals

• An indexing and abstracting service is a service that provides shortening or summarizing of documents and assigning of descriptors for referencing documents.

• Examples: – Index Copernicus

– Scopus

– Ulrichs Web

– ISI

– Australian Education Index

– DOAJ

– www.africaknowledgeproject.org/akpjournalindexed.htm

Levels of Research

Publication

Public Presentation

Write reports

Our work is reviewed by a

panel of experts

Our work is evaluated whether we are

contributing to theory

Levels of Publication

ISI abstracted Journals

Abstracted journal

International Journal

Local journal

Conference proceeding

In house publication

What does it mean to publish?

Studies that will be published should have Significant contributions

• New argument or

conjecture

• New definition

• Clarification

• illustration or exemplar

• Elaboration

• refutation or rebuttal

• rephrasing

• Rebuttal of question

• Recasting of question

• Evaluation of an earlier

assertion

• New or alternative

interpretation

• Supportive evidence

• Contrary evidence

Concern on the authors impact factor

• The h-index quantifies the actual scientific productivity and the apparent impact of the scientist. The h-index is based on the author’s most cited papers and the number of citations they have received from other articles.

• "A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np − h) papers have no more than h citations each." [For details in calculation, see Hirsch, 2005]

• An h-index of 16 means, for example, that a researcher has published 16 papers that each had at least 16 citations. Therefore, the h-index reflects both the number of articles as well as the number of citations per article.

• Google scholar • Web of science • Harzing’s Publish or Perish

Recommendations

• Review institutional research polices through various representations

• Guidelines in thesis and dissertation writing should be based on international standards on publication

• Libraries need to invest on access to high impact journals

• Get faculty who published in abstracted journals to teach research courses

• Train faculty in the discipline to teach statistics courses