Chapter 3

Quality criteria, research ethics, and other research issues

3.I Quality criteria for researchAs we have seen, the basic definition of scientific research is that

it is a 'disciplined' inquiry, and therefore one thing research cannotafford is to be haphazard or lacking rigour. Unfortunately, generalagreement about research quality in scholarly circles stops at therecognition of its importance; when it comes to specifying theconcrete 'quality criteria' to be applied, the literature ischaracterized by a host of parallel or alternative views and verylittle consensus.

3. I. I Quality criteria in quantitative researchThe concept of 'reliability‘ is fairly straightforward, but when welook at 'validity' we find two parallel systems in the quantitativeliterature-one centered around 'construct validity' and itscomponents, the other around the 'internal/external validity’dichotomy-and scholars tend to be surprisingly vague about therelationship between these two systems: the usual practice isthat a work either covers one or the other. This dualisticapproach is due to the fact that within the quantitative paradigmmeaningfulness' has been conceptualized from twoperspectives: research design and measurement.

The discussion of quantitative quality standards is bestdividedinto three parts: (a) reliability (b) measurement

validity, and(c) research validity.

ReliabilityThe term reliability comes from measurement theory and refersto the 'consistencies of data, scores or observations obtainedusing elicitation instruments, which can include a range of toolsfrom standardized tests administered in educational settings totasks completed by participants in a researchstudy.

Measurement validitythe concept of validity from a measurement perspectivehas traditionally been summarized by the simple phrase: a test is validif it measures what it is supposed to measure. Lynch (2003: 149)summarizes the new conception clearly: 'When examining the validityof assessment, it is important to remember that validity is a propertyof the conclusions, interpretations or inferences that we draw from theassessment instruments and procedures, not the instruments and

procedures themselves.

Research validityThe second type of validity, 'research validity' is broader thanmeasurement validity as it concerns the overall quality of the wholeresearch project and more specifically (a) the meaningfulness of theinterpretations that researchers make on the basis of theirobservations, and (b) the extent to which these interpretationsgeneralize beyond the research study (Bachman 2004a).

3.1.2 Quality criteria in qualitative researchThe usual statement we find in the literature about quality inQUAL research is that it is less straightforward to define thanquality in QUAN research. We saw in chapter 2 that a qualitativestudy is inherently subjective, interpretive as well as time andcontext bound; that is in a qualitative inquiry ‘truth’ is relativeand ‘facts’ depend upon individual perceptions(Morse andRichards 2002); for this reason several researchers have arguedthat qualitative research requires its own procedures forattaining validity that are different from those used inquantitative approaches.

Three basic quality concerns in qualitative research

1- Insipid data focusing on 'individual meaning' does not offer anyprocedures for deciding whether the particular meaning is interestingenough.

2- Quality of the researcher Morse and Richards (2002) are-right whenthey warn us that any study is only as good as the researcher, and in aqualitative study this issue is particularly prominent because in a waythe researcher is the instrument.

3- Anecdotalism and the lack of quality safeguards.

Strategies to ensure validity in qualitative research

Building up an image of researcher integritythe most important strategy to ensure the trustworthiness of a projectis to create in the audience an image of the researcher as a scholarwith principled standards and integrity. This image of integrity is madeup of several small components but there are certain strategies thatare particularly helpful in showing up the researcher's high standards(provided, of course, those exist):

• Leaving an audit trail By offering a detailed and reflective account ofthe steps taken to achieve the results.• Contextualization and thick description Presenting the findings in richcontextualized detail helps the reader to identify with the project andthus come on board.

• Identifying potential researcher bias Given the important role of theresearcher in every stage of a qualitative study.• Examining outliers extreme or negative cases and alternativeexplanations No research study is perfect and the readers know this.Therefore, explicitly pointing out and discussing aspects of the studythat run counter to the final conclusion is usually not seen as a

weakness but adds to the credibility of the researcher.

3. I . 3 Quality criteria in mixed methods researchThe rationale for mixing methodsThe important point to emphasize is that a mixed methods inquiryoffers a potentially more comprehensive means of legitimizing findingsthan do either QUAL and QUAN methods alone by allowing

investigators to assess information from both data types.The 'design validity' of the studythe term 'design validity‘ refers to the extent to which the QUALand QUAN components of a mixed methods study are combined orintegrated in a way that the overall design displays complementarystrengths and nonoverlapping weaknesses of the constituent methods.

3.2 Research ethicsSocial research-including research in education-concerns people's livesin the social world and therefore it inevitably involves ethical issues. AsPunch (2005) points out, such issues are more acute in QUAL than inQUAN approaches because qualitative research often intrudes moreinto the human private sphere: it is inherently interested in peoplespersonal views and often targets sensitive or intimate-matters.

3.2.2 Legal contextIn many countries, observing ethical principles is enforced by legal andinstitutional requirements. In the US, for example, researchers have tosubmit a detailed research plan for approval to an Institutional ReviewBoard prior to starting their investigations in order to comply withfederal regulations that provide protection against human rightsviolations.

3. 2. 3 Researcher integrityThe Ethical standards of the American Educational ResearchAssociation (AERA 2002), starts out with a set 'of 'guiding standards'describing the researchers' general responsibilities to the field. Theseinclude the following points:

- Educational researchers must not fabricate, falsify, or misrepresentauthorship, evidence, data, findings, or conclusions.- Educational researchers must not knowingly or negligently use theirprofessional roles for fraudulent purposes.- Educational researchers should attempt to report their findings to allrelevant stakeholders, and should refrain from keeping secret orselectively communicating their findings.

3.2.4 Protection from harmThe primary principle of research ethics is that no mental or physicalharm should come to the respondents as a result of their participationin the investigation.

3.2.5 Privacy, confidentiality and anonymityIt is a basic ethical principle that the respondent's right to privacyshould always be respected and that respondents are within theirrights to refuse to answer questions or to withdraw from the studycompletely without offering any explanation. t is also the participants'right to remain anonymous, and if the participant's identity is knownto the research group, it is the researcher's moral and professional(and in some contexts legal) obligation to maintain the level of

confidentiality that was promised at the onset.

3.2.6 Informed consent and the issue of deceptionThe most salient and most often discussed aspect of research ethics is

the issue of informed consent. In the US, for example, federalregulations not only require written consent from theparticipants but also require informed consent before aresearcher can use an individual's existing records for

researchpurposes ( Johnson andChristensen 2004).

DeceptionIt does not require much justification that sometimesresearchers cannot provide full disclosure of the nature andpurpose of the study without causing participant bias or eveninvalidating the study, and in some (rare) cases the researcherneeds not only to withhold some information but to activelymislead the participants.

Research topic, research purpose and research questionsEvery investigation has a starting point and unless we adopt someoneelse's (for example, the supervisor's) design idea or join an ongoinginvestigation as a co-researcher, this starting point is a broad 'researchtopic'.

The research purpose is relatively short statement that describes theobjective of the planned study, explaining why the investigation isundertaken and what its potential significance is.

what is a good research question like? When we have thought of somepossible questions, we have to ask ourselves: are they worth askingand, more importantly, answering? I believe that one thing we musttry and avoid at all cost is to 'run into a "so what“ response to ourresearch' (Mackey and Gass 2005: 17).

Other essentials for launching a study: pilot study,research log, and data management

piloting the researchPiloting is more important in quantitative studies than in qualitativeones, because quantitative studies rely on the psychometric

propertiesof the researchinstruments.

Research logAs with any real logbook, all the entries should be properly dated andthe consecutive pages of the logbook should be numbered and kepttogether in a folder. Alternatively, we can keep an electronic log file inour computer that we regularly update.

Techniques to manage and store data recordsResearchers need to make a detailed list of all the data sources in aresearch log (see previous section), organized according to keyparameters (for example, type of data, place of data-gathering) and then

need to keep these records up-to-date.