REVIEW : KEY RESEARCH CONCEPTS/TERMS - See Glossary at course website for sample definitions.

applied research, basic research=pure research exploratory, descriptive, explanatory, predictive research evaluation , research, evaluation research concepts, hypotheses, propositionsreactivity, ecological fallacy, causality deductive and inductive reasoning, variables, dependent, independent variables ways of knowing: science, common sense, experience, traditional knowledge research design cross sectional, longitudinal research , panel study, trend study qualitative, quantitative research unit of analysis nominal and operational definitions measurement validity, reliability ; random and systematic error (bias) continuous and discrete variables nominal, ordinal, interval, ratio measurement scales population, sample ; probability, non-probability samples, sampling frame purposive, quota, judgement, snowball, ...sampling simple random sample, systematic sample stratified, area sampling sampling error ; standard error of the mean statistic, parameter survey, questionnaire, interview pilot study, pretest experiment, observation closed, open ended questions, probes response rate ; non-response bias evaluability assessment evaluation research formative, summative evaluation, process evaluation needs assessment weighted average coding, cleaning of data descriptive and inferential statistics univariate statistics- frequency, mean, median, standard deviation, variance confidence interval, null hypothesis, significance level bivariate statistics- crosstab, correlation, comparison of means , chi square, t-test, ANOVA experiment, experimental group, control group, treatment pre-test, post-test with control design experimental errors - NB- premeasurement, interaction, history, selection * informed consent, anonymity vs confidentiality, UCRIHS *ethical issues in research & evaluation

1

REVIEW - KEY CONCEPTS/ POINTS 1. Distinguishing scientific from other ways of knowing/evaluating. 2. Types/categories of research 3. Steps in research process - general problem solving framework. Define problem, objectives, methods,

data gathering, analysis, conclusions, communication/ implementation. 4. General research designs/data gathering approaches. On-site designs only capture users/visitors;

household designs for general populations. Surveys to describe, experiments to study causal relationships, longitudinal to study trends/change over time.

5. Role of concepts, hypotheses, theory in research. 6. Operational definitions= systematic measurement procedures. 7. Reliability and validity of measures, random and systematic errors. 8. Levels or scales of measurement dictate which statistics are appropriate. 9. Variables, dependent, independent. Socio-economic, cognitive, affective and behavioral information. 10. Define study population first, then sampling approach. Sample statistics estimate population

parameters. 11. Probability vs nonprobablity sampling. 12. Key characteristic of sample is its representativeness of the population from which it is drawn. Rely on

laws of probability or judgement to draw sample. 13. Sampling error can be estimated for probability samples. Varies with size of sample, & homogenity of

population. 14. Stratify sample to more heavily sample subgroups with higher variance or assure a given number of

cases within each subgroup. Cluster sample to lower data gathering costs. 15. Pros and cons of mailed surveys, telephone and personal interviews. 16. Questionnaire design issues 17. Pretests, cover letters, follow-ups, nonresponse bias and related issues in survey research. 18. Qualitative vs quantitive approaches. 19. Problems suitable for observational approaches. 20. Basics of hypothesis testing and confidence intervals. Descriptve and inferential statistics. Comparing

means, crosstabs/chi square21. Experimental design - understand the pre-test-post-test with control design and how it controls for each

of the kinds of experimental errors. 22. Research ethics - informed consent, truth in reporting, confidentiality issues, not use research as guise

for selling. 23. Reporting - write/speak to the intended audience; executive summaries, oral presentations, technical

reports, proposals , tables and figures. 24. Examples of typical applications of various research designs/approaches Study tips

1. Read through the lecture topic outlines 2. Skim Powerpoint presentations3. Try the sample exam questions and check answers. 4. Make sure you can define each of terms in the list above/glossary, know what the term relates to

(where it fits in course), and can distinguish between groups of related terms. 5. Re-read selected readings in Trochim, your other text, or course links

Recommended : Leones – Guide to designing and conducting surveys. ASA Series on Surveys, U Wisconsin evaluation series.

2

SAMPLE EXAM QUESTIONS A. TRUE/FALSE ___ 1. Only researchers need to be knowledgeable about research and evaluation methods, as

managers and recreation practitioners can hire consultants to do whatever research may be needed.

___ 2. Researchers assess the validity of measurements by repeating the measurement to see if they get the same results.

___ 3. The measurement of importance and satisfaction in the Huron Clinton Metroparks Visitor Study were ordinal measures.

___ 5. Race/ethnic status is usually measured at an ordinal scale. ___ 6. In a mail survey, identification numbers are included on the survey so that the investigator can

identify who has returned a questionnaire. If the subject's identity is never associated with their individual responses in any reporting of results, the subjects are anonymous.

___ 7. To test a relationship between two nominal scale variables, one could use a correlation. ___ 8. The chi square statistic tests for differences in means of two subgroups. ___ 9. When testing at the 95% confidence level, one should reject the null hypoythesis if the reported

significance level is greater than .05. ___ 10. Inferential statistics describe the characteristics of a sample. B. MULTIPLE CHOICE. ___ 1. Which of the following is a reason for using an outside consultant to conduct a survey for a recreation organization: a) they know the organization best, b) they know technical details of conducting a survey, c) they can do the study more cheaply, d) they are more knowledgeable about your customers.

___ 2. The first step in a research study is a) reviewing previous studies b) determining the sample size, c) designing the questionnaire, d) identifying the study's purpose.

___ 3. Applied research studies generally a) have a specific client for the research, b) have a fairly immediate time frame for using the results, c) seek to advance theories of human behavior, d) both a and b, e) all of the above.

___ 4. The most important factor in determining the research design for a given problem is a) the designs and methods the researcher is most familiar with, b) the nature of the problem, c) the designs scientists have used in the past, d) the methods the client feels comfortable with.

___* 5. Your text, Applied Social Research, claims that science is provisional. By this they mean a) it is based on observations in the real world, b) uses inductive and deductive logic, c) is self-corrective, or d) is wishy-washy in drawing conclusions.

___ 6. If we attempted to describe visitors to a park based on a very small random sample (say 50 people), our estimates would likely be a) unreliable, b) invalid, c) both of the above, d) neither of the above.

___ 7. If we attempt to describe visitors to a park across the entire year by using a large summer season survey, our estimates would likely be a) unreliable, b) invalid, c) both of the above, d) neither of the above.

___ 8. The fundamental tradeoff in most research design is between accuracy of results and a) protecting the rights of subjects, b) time and costs of the study, c) usefulness of results, d) level of detail.

___ 9. Crystal Mt. Ski Area conducted a lift line survey to determine where visitors came from, how often they skied last year, and how many times they had been to Crystal Mt. last winter. This study is an example of a(n) a) exploratory, b) descriptive, c) explanatory, or d) predictive study.

3

__ 10. The Rockville Parks Department wishes to increase the number of Black and Hispanic visitors to its parks. It uses a nominal group process to brainstorm ideas and select the most promising options to work on. The nominal group process is followed by focus groups to flesh out and evaluate each alternative before deciding which to implement. This study is an example of a(n) a) formative, b) summative, c) process, or d) outcome evaluation study. C. FILL IN BLANKS. Fill in the blanks .

1. The _________________ variable in a study is generally the one we wish to predict or explain.

2. A(n) ___________ is a scientific statement that predicts or specifies a relationship between two or more variables and is tested through research.

3. The prefered approach to identify cause-effect relationships is ___________________. D. Short Answer 1a. List the four levels of measurement in order of increasing power & precision. Give an example of a variable that would usually be measured at each level. Then place each of the following statistics next to the level to which it best applies. (Statistics: median, frequency, mode, mean, range, standard deviation, inter-quartile deviation).

Measurement Level Example Appropriate Statistics_______________ _________________ ________________________________ _________________ ________________________________ _________________ ________________________________ _________________ _________________

2.. Distinguish between each of the following pairs of concepts:

a. reliability and validity of a measureb. probability and nonprobability samplec. descriptive and inferential statisticsd. a parameter and a statistice. a survey and an experimentf. independent and dependent variableg. anonymity and confidentialityh. standard deviation and standard errori. pre-measurement (testing), history, and interaction errors in an experiment

3. You have been given a research report that presents the methods and findings from a study evaluating the image of Michigan as a travel destination in the surrounding five state region. Identify the three most important things you would look for in the report to assess the credibility of the findings.

4 . The manager of Sleepy Hollow State Park about 20 miles northeast of Lansing wants to identify the market area from which the majority of current visitors come. He also wants to be able to compare campers and day users and determine what proportion of each participates in fishing at the park. The park has a year to complete the study.

a. Write ONE research objective for this study. b. Define a suitable study population for this study (element, unit, extent, time). c. Recommend the overall research approach (i.e. choose the location (household, on-site, lab), ... and

data gathering approach (observation, phone, personal interview, etc.).d. Based on the brief problem statement , identify four important variables to measure in the study.

4

e. What would be your greatest concern (potential problem or source of error) in designing this study?

5. The PRR 844 instructor wishes to use an experiment to evaluate the contribution of micro-computer labs to performance in the class. Next year the class will be divided into two sections, one with a computer lab and one without. Students research/evaluation skills will be evaluated using an exam given before and after the class.

a. Diagram the experimental design being used here, b. Identify the treatment, experimental group, control group, and the measure of the effect. c. How would you assign students to the two groups.d. If the experimental group's scores increase from 35 to 70 and the control group's increase from

35 to 50, what is the "effect" of the computer labs?

6. a. A survey of 200 participants in a fitness program finds that 30% would enroll in the program again. . Using the table for sampling errors for a binomial distribution (attached) , compute a 95% confidence interval for this estimate.

Sampling Errors for a binomial distribution (95% confidence interval)

Distribution in the populationSample size 50/50 60/40 70/30 80/20 90/10100 10.0% 9.8% 9.2% 8.0% 6.0%200 7.1% 6.9% 6.5% 5.7% 4.2%400 5.0% 4.9% 4.6% 4.0% 3.0%1000 3.2% 3.1% 2.9% 2.5% 1.9%1500 2.6% 2.5% 2.4% 2.1% 1.5%2000 2.2% 2.2% 2.0% 1.8% 1.3%

b. In a survey of 100 randomly chosen downhill skiers, the average spending on the trip was $500. If the standard deviation in the sample was $400, compute a 95% confidence interval for the spending estimate in the population.

7. For each of the following research problems, choose what you feel is the best approach (from the list) Briefly identify the study population and explain why you chose the given design.

Designs to choose from

Household survey (Telephone, Mailed, or Personal Interview)On-site survey (self-administered or personal interview)ObservationExperimentSecondary data analysisQualitative approach (focus group, in depth interview or case study)

a. MSU wishes to assess the percentage of bicyclists on campus wearing helmets. b. A Lake Michigan marina wants to assess the potential market for a new full service marina.c. A Lake Michigan marina wants to see how many of its existing slip holders would stay with the

marina if it raises its rates by 20% next year.d. East Lansing wishes to determine community attitudes about a proposed new swimming facility.e. MSU wants to determine if SAT scores are a good predictor of performance in college, as judged by

grade point averages.

5

ANSWERS TO SAMPLE EXAM:A. 1. F – managers must understand enough about research to communicate needs to researchers,

participate in studies and interpret and evaluate reports.2. F – reliability, not validity refers to repeatability of measures3. T – most attitude scales including very important, somewhat, … etc are ordinal5. F – race/ethnicity is nominal ; unordered categories6. F - confidential if we can associate responses with an individual, but don’t7. F – correlations are for interval scale, for nominal use chi square test8. F – chi square tests for patterns in a table, F or t-tests (compare means) compare means.9. F – reject null hypothesis if SIG < .05. Think of the significance level (.05) as the probability the sample

could look like this if null hypothesis were true. Small probability leads us to reject the null hypothesis.

10. F – descriptive statistics describe; inferental stats test hypotheses/infer to population.

B. 1. B , 2. D, 3. D , 4. B, 5. C , 6. A, 7. B, 8. B, 9, B, 10. A C. dependent, hypothesis, experimentD. 1. see lecture outlines page 34, examples race (N), any attitude (SA, A, N D, SD) is ordinal or any

interval characteristic measured in broad categories, age, height, weight, days participated in an activity, amount spent are interval.

2. a. reliability measures repeatability of a measure (random error), validity measures absence of systematic errors or bias – are we measuring what we think?

b. probability –each person has known chance of selection, otherwise non-probc. descriptive stats describe characteristics of sample (freq, avg), while inferential stats test

hypotheses or generalize from sample to population. d. statistic is summary measure of a variable in sample; parameter,in a population.e. survey measures things as they are, experiment manipulates at least one variable to test effect on

others.f. dependent varable is one you want to explain or predict, independent vars are the ones which do

the explaining.g. anonymity – can’t identify respondents, confidential if can but won’t.h. standard dev is measure of variation (spread) of a variable in a population; standard error (SE)

is the standard deviation of sampling distribution = spread of means for different samples of same size drawn at random from a given population. SE = STD Dev/ sqrt(n) , where n is sample size.

i. these are errors that can occur in an experiment; pre-measurement is the effect of a pre-test on post-test, history covers changes in any other variable besides the treatment that could cause a change in the effect, interaction captures an exaggerated or dampened effect of the treatment due to an interaction with the pre-test.

3. Some good answers are a) how big a sample, b) how did they measure image (reliable and valid?), c) is sample representative of intended population, d) what was the response rate (non-response bias?), e) who did the study and who was the sponsor, f) is report written clearly and objectively, g) use phone, mail, or personal interview.

4. Sample answers, others are possiblea. Determine market area of the park ; Compare activities of campers and day users – note thesee follow

directly from problem statement.b. Individuals (16 years and older) in vehicles leaving Sleepy Hollow State park between Jan 1 and Dec 31,

2002.c. On-site survey, short personal interview at exit gate: ask zipcode of residence, whether they were

camping, and what activities they did on this visit.d. zipcode of residence, camping or not on this visit, participate in fishing, maybe gender, bring a boat?e. main problem will be obtaining representative sample across time of day, day of week and season.

Stratify sample by these time periods and use use counts to adjust to total population.

5. Standard pre-test /post-test with control group

6

a. DiagramMBe X MAe - experimental groupMBc MAc - control group

b. treatment is the micro-lab (X), measure of effect is change in experimental group – change in control group

= (MAe-MBe ) - ( MAc-MBc )

c. Randomly assign to two groups by picking every other student on alphabetical class list to be in experimental group.d. (70-35) – (50-35) = 35-15 = 20 - twenty point increase due to micro-lab

6a. sampling error is 6.5% (n=200, and the 70/30 column) so 95% CI is 30% plus or minus 6.5% = ( 23.5, 36.5) 6b. standard error is standard deviation/sqrt(n) = 400/sqrt(100) = 400/10=40 95% conf interval is two standard errors either side of mean = $500 plus or minus $80

= ($420, $580)7. Often more than one reasonable alternative. Key is explaining your answer.a. observation, whenever variable is easily observed; pick random locations and times to observe.b. household survey to a list of registered boaters (large boats within X miles) or possibly an “on-site”

survey of boaters at nearby marinas; might also suggest a set of focus groups of current large boat owners/marina slip renters in the area for a more qualitative assessment.

c. On-site survey of current slip renters at the marina, experiment would be bolder but likely not feasible/wise, e.g. send out notices of price increases to half of slip renters and ask who plans to return. Compare with other half who are given the current rate.

d. household telephone survey of EL residents for quantitative estimates; or a set of focus groups, community forums/workshops for qualitative assessment of attitudes.

e. secondary data using records of students for the past five years – inexpensive.

7

Glossary of Research and Evaluation Terms (from Monette, Sullivan and DeJong)

Anonymity: a situation in which no one, including the researcher, can link individual's identities to their responses or behaviors that serve as research data.

Applied research: designed with a practical outcome in mind and with the assumption that some group or society as a whole will gain specific benefits from the research.

Area Sampling: a multistage sampling technique that involves moving from larger clusters of units to smaller and smaller ones until the unit of analysis, such as the household or individual, is reached. (Cluster sampling). Sampling by dividing the population into groups or clusters and drawing samples from only some groups.

Available Data: observations collected by someone other than the investigator for purposes that differ from the investigator's but that are available to be analyzed. (Secondary data)

Baseline: a series of measurements of a client's condition prior to treatment that is used as a basis for comparison with the client's condition after treatment is implemented.

Basic Research: research conducted for the purpose of advancing knowledge about human behavior with little concern for the immediate or practical benefits that might result.

Bivariate Statistics: statistics that describe the relationship between two variables.Blocking: a two stage system of assigning subjects to experimental and control groups whereby subjects

are first aggregated into blocks according to one or more key variables; members of each block are then randomly assigned to experimental and control groups.

Causality: the situation where an independent variable is the factor - or one of several factors - that produces variation in a dependent variable.

Closed Ended Questions: questions that provide respondents with a fixed set of alternatives from which they are to choose.

Coding: the categorizing of behavior into a limited number of categories. Common Sense: practical judgments based on the experiences, wisdom, and prejudices of a people.Concepts: mental constructs or images developed to symbolize ideas, persons, things, or events.Concurrent Validity: a type of criterion validity in which the results of a newly developed measure are

correlated with results of an existing measure.Confidentiality: ensuring that information or responses will not be publicly linked to specific individuals

who participate in research.Construct Validity: a complex approach to establishing the validity of measures involving relating the

measure to a complete theoretical frame work, including all the concepts and propositions that the theory comprises.

Content Analysis: a method of transforming the contents of documents from a qualitative, unsystematic form to a quantitative, systematic form.

Content Validity: an approach to establishing the validity of measures involving assessing the logical relationship between the proposed measure and the theoretical definition of the variable.

Continuous Variables: variables that theoretically have an infinite number of values.Control Group: the subjects in an experiment who are not exposed to the experimental stimulus.Control Variables: variables whose value is held constant in all conditions of an experiment.Convenience Samples: samples composed of those elements that are readily available or convenient to the

researcher.Cost-Benefit Analysis: an approach to program evaluation wherein program costs are related to program

benefits expressed in dollars.Cost-Effectiveness Analysis: an approach to program evaluation wherein program costs are related to

program effects, with effects measured in the units they naturally occur.Cover Letter: a letter that accompanies a mailed questionnaire and serves to introduce and explain it to the

recipient.Criterion Validity: a technique for establishing the validity of measures that involves demonstrating a

correlation between the measure and some other standard.Double Blind Experiment: an experiment conducted in such a way that neither the subjects nor the

experimenters know which groups are in the experimental and which are in the control condition.Ecological Fallacy: inferring something about individuals from data collected about groups.Ethics: the responsibilities that researchers bear toward those who participate in research, those who

sponsor research, and those who are potential beneficiaries of research.

8

Cross Sectional Research: research based on data collected at one point in time.Data Analysis: the process of placing observations in numerical form and manipulating them

according to their arithmetic properties to derive meaning from them.Data Archives: a national system of data libraries that lend sets of data, much as ordinary libraries lend

books.Deductive Reasoning: inferring a conclusion from more abstract premises or propositions.Descriptive Research: research that attempts to discover facts or describe reality.

Descriptive Statistics : procedures that assist in organizing, summarizing, and interpreting the sample data we have at hand.

Dimensional Sampling : a sampling technique designed to enhance the representativeness of small samples by specifying all important variables and choosing a sample that contains at least one case to represent all possible combinations of variables.

Direct Costs: a proposed program budget or actual program expenditures.Discrete Variables: variables with a finite number of distinct and separate values. Evaluability Assessment: a preliminary investigation into a program prior to its evaluation to determine

those aspects of the program that are evaluable.Evaluation Research: the use of scientific research methods to plan intervention programs, to monitor the

implementation of new programs and the operation of existing programs, and to determine how effectively programs or clinical practices achieve their goals.

Experiential Knowledge: knowledge gained through firsthand observation of events and based on the assumption that truth can be achieved through personal experience.

Experimental Group: those subjects who are exposed to the experimental stimulus or treatment.Experimental Stimulus or Treatment: the independent variable in an experiment that is manipulated by the

experimenter to assess its effect on behavior.Experimental Variability: variation in a dependent variable produced by an independent variable.Experimentation: a controlled method of observation in which the value of one or more independent

variables is changed in order to assess its causal effect on one or more dependent variables.Explanatory Research: research with the goal to determine why or how something occurs.External Validity: the extent to which causal inferences made in an experiment can be generalized to other

times, settings, or peopleExtraneous Variability: variation in a dependent variable from any source other than an experimental

stimulus.Face Validity: another name for content validity. See Content Validity.Field Experiments: experiments conducted in naturally occurring settings as people go about their

everyday affairs.Field Notes: detailed, descriptive accounts of observations made in a given setting.Formative Evaluation Research: evaluation research that focuses on the planning, development, and

implementation of a program.Fraud (scientific): the deliberate falsification, misrepresentation, or plagiarizing of data, findings, of the

ideas of others.Grant: the provision of money or other resources to be used for either research or service delivery

purposes.Guttman Scale: a measurement scale in which the items have a fixed progressive order and that has the

characteristic of reproducibility.Human Services: professions with the primary goal of enhancing the relationship between people and

societal institutions so that people may maximize their potential.Hypotheses: testable statements of presumed relationships between two or more concepts.Independent Variable: the presumed active or causal variable in a relationship.Index: a measurement technique that combines a number of items into a composite score.Indicator: an observation assumed to be evidence of the attributes or properties of some phenomenon.Inductive Reasoning: inferring something about a whole group or class of objects from knowledge of one

or a few members of that group or class.Inferential Statistics: procedures that allow us to make generalizations from sample data to the

populations from which the samples were drawn.Informed Consent: telling potential research participants about all aspects of the research that might

reasonably influence their decision to participate.

9

Internal Validity: an issue in experimentation concerning whether the independent variable actually produces the effect it appears to have on the dependent variable.

Interval Measures: measures that classify observations into mutually exclusive categories with an inherent order and equal spacing between the numbers produced by a measure.

Interview: a technique in which an interviewer reads questions to respondents and records their verbal responses.

Interview Schedule: a document, used in interviewing, similar to a questionnaire, that contains instructions for the interviewer, specific questions in a fixed order , and transition phrases for the interviewer.

Item: a single indicator of a variable, such as an to a question or an observation of some behavior or characteristic.

Judgmental Sampling: a non-probability sampling technique in which investigators use their judgement and prior knowledge to choose people for the sample who best serve the purposes of the study.

Measurement Scale: a measurement device allowing responses to a number of items to be combined to form a composite score on a variable.

Laboratory Experiments: experiments conducted in artificial settings constructed in such a way that selected elements of the natural environment are simulated and features of the investigation are controlled.

Measures of Association: statistics that describe the strength of relationships between variables.Levels of Measurement: rules that define permissible mathematical operations on a given set of numbers

produced by a measure. See Nominal, ordinal, interval and ratio.Likert Scale: a measurement scale consisting of a series of statements followed by five response

alternatives, typically: strongly agree, agree, no opinion, disagree, or strongly disagree.Longitudinal Research: research based on data gathered over an extended time period.Matching: a process of assigning subjects to experimental and control groups in which each subject is

paired with a similar subject in the other group.Measurement: the process of describing abstract concepts in terms of specific indicators by the as-

signment of numbers or other symbols to these indicants in accordance with rules.Measurement Scale: a measurement device allow responses to a number of items to be combined to form a

composite score on a variable.Measures of Association: statistics that describe the strength of relationships between variables.

Measures of Central Tendency: statistics, also known as averages, that summarize distributions of data by locating the "typical" or "average" value.

Measures of Dispersion: statistics that indicate how dispersed or spread out the values of a distribution are.Misconduct (scientific). scientific fraud, plus such activities as carelessness or bias in recording or

reporting data, mishandling data, and incomplete reporting of results.Missing Data: incomplete data found in available data sets.Multidimensional Scaling: a scaling technique designed to measure complex variables composed of more

than one dimension.Multistage Sampling: a multiple tiered sampling technique that involves moving from larger clusters of

units to smaller and smaller ones until the unit of analysis, such as the household or individual, is reached.

Multi-trait Multi-method Approach to Validity: a particularly complex form of construct validity involving the simultaneous assessment of numerous measures and numerous concepts through the computation of inter-correlations.

Multivariate Statistics: statistics that describe the relationships among three or more variables.Needs Assessment: collecting data to determine how many people need particular services and to assess

the level of services or personnel that already exist to fill that need.Nominal Definitions: verbal definitions in which one set of words or symbols is used to stand for another

set of words or symbols.Nominal Measures: measures that classify observations into mutually exclusive categories but with no

ordering to the categories.Non-probability Samples: samples in which the probability of each population element being included in

the sample is unknown.Non-reactive Observation: observation in which those under study are not aware that they are being

studied and the investigator does not change their behavior by his or her presence.Observational Techniques: the collection of data through direct visual or auditory experience of behavior.

10

Open Ended Questions: questions without a fixed set of alternatives, which leaves respondents completely free to formulate their own responses.

Operational Definitions: definitions that indicate the precise procedures or operations to be followed in measuring a concept.

Opportunity Costs: the value of foregone opportunities incurred by funding one program as opposed to some other program.

Ordinal Measures: measures that classify observations into mutually exclusive categories that have an inherent order to them.

Panel Study: research in which data are gathered from the same people at different times.Parameter : summary description of a variable in a population.Participant Observation: a method in which the researcher is a part of, and participates in, the activities of

the people, group, or situation that is being studied.Physical Traces: objects or evidence that result from people's activities that can be used as data to test

hypotheses.Pilot Study: a trial run on a small scale of all procedures planned for a research project.Population: all possible cases of what we are interested in studying.Positivism: the perspective that human behavior should be studied only in terms of behavior that can be

observed and recorded by means of some objective technique.Predictive Research: research that attempts to make projections about what will occur in the future or in

other settings.Predictive Validity: a type of criterion validity wherein scores on a measure are used to predict some

future state of affairs.Pre-experimental Designs: crude experimental designs that lack the necessary controls of the threats to

internal validity.Pretest: a preliminary application of the data gathering technique to assess the adequacy of the technique.Privacy: the ability to control when and under what conditions others will have access to your beliefs,

values, or behavior.Probability Samples: samples in which each element in the population has a known chance of being

selected into the sample.Probes: follow-up questions used during an interview to elicit clearer and more complete responses.Propositions: statements about the relationship between elements in a theory.Pure Research: research conducted for the purpose of advancing our knowledge about human behavior

with little concern for any immediate or practical benefits that might result.Purposive Sampling: a non-probability sampling technique wherein investigators use their judgment and

prior knowledge to choose people for the sample who would best serve the purposes of the study.Qualitative Research: research that focuses on data in the form of words, pictures, descriptions, or

narratives.Quantitative Research: research that uses numbers, counts, and measures of things.Quasi Experimental Designs: designs that approximate experimental control in non-experimental settings.Questionnaire: a set of written questions that people respond to directly on the form itself with out the aid

of an interviewer.Quota Sampling: a type of non-probability sampling that involves dividing the population into various

categories and determining the number of elements to be selected from each category.Random Assignment: a process for assigning subjects to experimental and control groups that relies on

probability theory to equalize the groups.Random Errors: measurement errors that are neither consistent nor patterned.Ratio Measures: measures that classify observations into mutually exclusive categories with an inherent

order, equal spacing between the categories, and an absolute zero point.Reactivity: the degree to which the presence of a researcher influences the behavior being observed.Reliability: the ability of a measure to yield consistent results each time it is applied.Representative Sample: a sample that accurately reflect the distribution of relevant variables in the target

population.Research Design: a detailed plan outlining how a research project will be conducted.Response Bias: responses to questions that are shaped by factors other than the person's true feelings,

intentions or beliefs.

11

Response Rate: the percentage of a sample that completes and returns a questionnaire or agrees to be interviewed.

Sample: one or more elements selected from a population.Sampling Error: the extent to which the values of a sample differ from those of the population from

which it was drawn.Sampling Frame: a listing of all the elements in a population.Scale: a measurement technique, similar to an index, that combines a number of items into a composite

score.Science: a method of obtaining objective knowledge about the world through systematic observation.Secondary data Analysis: the reanalysis of data previously collected for some other research project.Semantic Differential: a scaling technique that involves respondents rating a concept on a scale between a

series of polar opposite adjectives.Simple Random Sampling: a sampling technique wherein the target population is treated as a unitary

whole and each element has an equal probability of being selected for the sample.Single Subject Designs: quasi-experimental designs featuring continuous or nearly continuous

measurement of the dependent variable on a single research subject over a time interval that is divided into a baseline phase and one or more additional phases during which the in dependent variable is manipulated; experimental effects are inferred by comparisons of the subjects responses across baseline and intervention phases.

Snowball Sampling: a type of non-probability sampling characterized by a few cases of the type we wish to study leading to more cases, which, in turn, lead to still more cases until a sufficient sample is achieved.

Social Research: a systematic examination (or re examination) of empirical data collected by someone firsthand, concerning the social or psychological forces operating in a situation.

Standard deviation – a measure of spread or variation in a distribution, equals the square root of the average squared deviation from the sample mean. The standard deviation squareed is called the variance of the distribution.

Standard error of the mean : standard deviation of the sampling distribution. A 95% confidence interval around the estimate of the population mean is the sample mean plus or minus 1.96 * standard error. The standard error is the standard deviation in the population divided by the square root of the sample size.

Statistic: A summary description of a variable in a sample.Statistics: procedures for assembling, classifying, and tabulating numerical data so that some meaning or

information is derived. Stratified Sampling: a sampling technique wherein the population is subdivided into strata with separate

sub-samples drawn from each strataSummated Rating Scales: scales in which a respondent's score is determined by summing the numbers of

questions answered.Summative Evaluation Research: evaluation research that assesses the effectiveness and efficiency of

programs and the extent to which program effects are generalizable to other settings and populations.

Survey: a data collection technique in which information is gathered from individuals, called respondents, by having them respond to questions.

Systematic Errors: measurement errors that are consistent and patterned.Systematic Sampling: a type of simple random sampling wherein every nth element of the sampling frame

is selected for the sample.Theory: a set of interrelated propositions or statements, organized into a deductive system, that offers an

explanation of some phenomenon.Thurstone Scale: a measurement scale consisting of a series of items with a predetermined scale value

to which respondents indicate their agreement or disagreement.Time Sampling: a sampling technique used in observational research in which observations are made only

during specified pre-selected times.Traditional Knowledge: knowledge based on custom, habit, and repetition.Trend Study: research in which data are gathered from different people at different times.True Experimental Designs: experimental designs that utilize randomization, control groups, and other

techniques to control threats to internal validity.

12

Uni-dimensional Scale: a multiple item scale that measures one, and only one, variable.Units of Analysis: the specific objects or elements whose characteristics we wish to describe or explain

and about which data are collected.Uni-variate Statistics: statistics that describe the distribution of a single variable.Unobtrusive Observation: observation in which those under study are not aware that they are being

studied and the investigator does not change their behavior by his or her presence.Validity: the degree to which a measure accurately reflects the theoretical meaning of a variable.Variables: operationally defined concepts that can take on more than one value.Verification: The process of subjecting hypotheses to empirical tests to determine whether a theory is

supported or refuted.Verstehen: the effort to view and understand a situation from the perspective of the people actually in that

situation.

13