Methods for a Single Categorical Variable – Simulation

Example: Evaluating a claim of hearing lossConsider the case study presented in an article by Pankratz, Fausti, and Peed titled “A Forced-Choice Technique to Evaluate Deafness in the Hysterical or Malingering Patient.” Source: Journal of Consulting and Clinical Psychology, 1975, Vol. 43, pg. 421-422. The following is an excerpt from the article:

The patient was a 27-year-old male with a history of multiple hospitalizations for idiopathic convulsive disorder, functional disabilities, accidents, and personality problems. His hospital records indicated that he was manipulative, exaggerated his symptoms to his advantage, and that he was a generally disruptive patient. He made repeated attempts to obtain compensation for his disabilities. During his present hospitalization he complained of bilateral hearing loss, left-sided weakness, left-sided numbness, intermittent speech difficulty, and memory deficit. There were few consistent or objective findings for these complaints. All of his symptoms disappeared quickly with the exception of the alleged hearing loss.

To assess his alleged hearing loss, testing was conducted through earphones with the subject seated in a sound-treated audiologic testing chamber. Visual stimuli utilized during the investigation were produced by a red and a blue light bulb, which were mounted behind a one-way mirror so that the subject could see the bulbs only when they were illuminated by the examiner. The subject was presented several trials on each of which the red and then the blue light were turned on consecutively for 2 seconds each. On each trial, a 1,000-Hz tone was randomly paired with the illumination of either the blue or red light bulb, and the subject was instructed to indicate with which light bulb the tone was paired.

Suppose that the patient was asked to do this a total of 12 times. The possible outcomes for the number of correct matches the patient gives overall will range from 0 to 12.

Questions:

1. If the patient actually suffers from hearing loss, what percent of the time would you expect him to correctly match the tone and color?

2. Using the answer from Question 1, what is the expected number of correct matches if the patient actually suffers from hearing loss and is therefore guessing on each trial?

3. Are there other value(s) for the number of correct matches you would anticipate possibly observing if the patient is simply guessing on each trial? What are these values?

4. What value(s) would you anticipate observing if the patient were intentionally giving the wrong answer to make it look as though he couldn’t hear?

1

Number of Correct Matches

Statistics can be used to determine whether or not a patient is intentionally giving the wrong answers. To investigate this situation, we can __________________ the possible outcomes that a hearing impaired person (that is, the person is guessing each time) would give for 12 trials of this experiment. For each replication of this experiment, we will keep track of how many times the patient correctly matched the tone and color. Once we’ve repeated this process several times, we’ll have a pretty good sense for what outcomes would be ________ surprising, or __________________ surprising, or ________ so surprising if the patient truly is hearing impaired.

Questions:

5. How can we replicate a hearing impaired individual using this applet?

6. Next, carry out 12 trials that simulate the responses of a hearing impaired individual and record your results below.

Trial Choice Correctly Matched?1 Red Blue Yes No2 Red Blue Yes No3 Red Blue Yes No4 Red Blue Yes No5 Red Blue Yes No6 Red Blue Yes No7 Red Blue Yes No8 Red Blue Yes No9 Red Blue Yes No

10 Red Blue Yes No11 Red Blue Yes No12 Red Blue Yes No

How many correct matches did you have? __________________

Next, collect the simulation results from everybody in the class. Make a dot for your outcome and that of all others in your class on the following number line.

2

Number of Correct Matches

Questions:

7. Given the simulation results on the above dotplot, what would you think about the patient’s claim that he suffered hearing loss if he had ...

a. 5 correct matches?

b. 0 or 1 correct matches?

c. 2 or 3 correct matches?

8. In your opinion, on how many of the 12 trials would the patient have to match the tone and color in order for you to be convinced they were intentionally giving wrong answers?

9. Ask some of your neighbors at what point they would become convinced the patient was intentionally giving wrong answers.

Neighbor 1: ________ Neighbor 2: ________

Neighbor 3: ________ Neighbor 4: ________

10. What potential issues arise when different people use different values for the point at which they become convinced the subject is intentionally giving wrong answers? Explain.

3

Because simply guessing on each trial is a random process, we know that we won’t observe the same number of

correct matches with every repetition of the experiment. However, the _________________ of the number of

correct matches obtained by guessing on 12 trials does follow a predictable pattern. This pattern started to emerge

in the dotplot from the activity; however, to get a more accurate sense of this pattern we should simulate the

experiment over and over again, say 1,000 times. The following graphic shows one possibility of what the distribution

would look like with 1,000 repeated samples.

Questions:

11. What does each dot in the plot represent?

12. Suppose the patient was correct on 4 of the 12 trials. Would you believe they were probably guessing, even though they had fewer than the expected number of correct guesses? Explain.

13. Suppose the subject was incorrect on 1 of the 12 trials. Would you believe they were just guessing or that they were intentionally answering incorrectly in order to mislead the researchers into thinking they were deaf? Explain.

14. What about 2, 3, or 4 correct matches, etc.? How many correct answers must a subject have given before you become convinced they were intentionally giving the wrong answers? You should be using the simulation results plotted above to help you answer this question.

4

15. In Question 8 above, you gave your personal opinion of this cutoff point. This was based on only a few simulated experiments. Has your cutoff changed now that you have seen the result of 1,000 simulated experiments? What do you think the cutoff should be?

16. Ask some of your neighbors at what point they would now become convinced the subject was intentionally giving the wrong answers.

Neighbor 1: ________ Neighbor 2: ________

Neighbor 3: ________ Neighbor 4: ________

17. The patient was actually presented with 100 trials. If he truly suffered from hearing loss, how many trials would you expect him to correctly identify with which stimulus the tone was paired?

18. Suppose the patient correctly identifies with which stimulus the tone was paired in only 45 of the 100 trials. A researcher argues that since this was less than the expected number of correct answers that the patient is intentionally giving incorrect answers to convince them he can’t hear. What is wrong with this reasoning?

19. Suppose the patient correctly identifies with which stimulus the tone was paired in none of the 100 trials. A researcher believes this result provides evidence that the subject must be intentionally giving incorrect answers in order to convince them he can’t hear. Do you agree?

5

Once again, the key question is how to determine whether the subject’s result on the 100 trials is surprising under the assumption that he truly has hearing loss and is simply guessing on each trial. To answer this, we will simulate the process of guessing on 100 trials of this experiment, over and over again. Each time we simulate the process, we’ll keep track of how many times the subject was correct (note that you could also keep track of the number of times he was incorrect). Once we’ve repeated this process a large number of times, we’ll have a pretty good sense for what outcomes would be very surprising, or somewhat surprising, or not so surprising if the subject is really guessing.We will use the applet at http://www.rossmanchance.com/applets/OneProp/OneProp.htm to help us carry out the simulation study:

Setting up the Simulation

Because we are assuming the patient is guessing and has a 50% of guessing correctly, we need to set the Probability of heads: to 0.50 (which is the default).

Since the patient was presented with 100 trials, the Number of tosses: needs to be set to 100. This will simulate guessing on 100 trials of the red/blue light bulb experiment.

Click Draw Samples and you should get the results from 100 trials of the red/blue light bulb experiment. One possible simulation result is given below.

6

Question:

20. How many correct answers did you get? _____________

Collecting the Results from Many Runs of the SimulationIdeally, we want to carry out this simulated experiment many more times, say 1,000 total. We can change the Number of repetitions: in the applet to accomplish this. Each time, the number of _______________ matches (i.e. heads) will be recorded. Change the Number of repetitions: to 999 (since you’ve already collected one trial). Roughly sketch the final plot you obtained below:

Questions:

21. What does each dot in the plot represent?

22. Suppose a patient was correct on 43 out of the 100 trials. Would you believe they were probably just guessing, even though they had fewer than expected number of correct answers? Why or why not?

23. The actual patient was correct 36 of the 100 trials. Based on this statistical investigation, do you believe he was just guessing or do these results indicate that he may have been answering incorrectly on purpose in order to mislead the doctors into believing he was deaf? Explain your reasoning, and use the dotplot of the simulation results to defend your answer.

7

Example: Swordtail FishA study was conducted to determine whether female swordtail fish preferred a symmetrical sexual signal. The signal that was examined is a pattern of dark vertical bars on male swordtail fish that become darker when courting a female. The researchers tested whether the females preferred males with symmetrical bar numbers (the same number of bars on the right and left side) over males with asymmetrical bar numbers. The study was conducted as follows.

Before the experiment, pairs of males were freeze-branded so that both males had the same total number of bars. One male in the pair, however, was randomly assigned to receive an asymmetrical number of bars (two more bars on one side than on the other, as shown below). The other male in the pair received a symmetrical number of bars (the same number on each side). Researchers randomly determined which side of the male would have more bars for the asymmetrical treatments. Then, all three fish were placed in the same aquarium, and the total time females spent with each male fish was recorded. The researchers recorded whether or not each female spent more time associating with the male that was symmetrical, thus showing a preference. The experiment was carried out with 6 female fish, so there were six trials.

Source: Morris and Casey. 1998. “Female swordtail fish prefer symmetrical sexual signal.” Animal Behavior, 55, 33-39.

Questions:

24. Suppose for a moment that the researcher’s conjecture is incorrect, and female swordtail fish do not really show any preference for either the symmetric or asymmetric male. In other words, the female swordtail fish just blindly pick one male or the other, without regard for whether he was symmetric. How many of the female swordtail fish would you expect to pick the symmetric male? Explain your reasoning.

25. At what point would you start to believe the female swordtail fish prefer the symmetric male?

8

Number of Females that Chose Symmetric

Note that the even if the observed number of symmetric male swordtail fish chosen is larger than would be expected, this is not necessarily enough statistical evidence to support the female fish’s preference for a symmetric male.

A key question is how to determine whether the study’s result is surprising under the assumption that female swordtail fish have no preference and are blindly picking a male fish.

Setting up the simulation

Questions:

26. What should the Probability of heads: be set to? Why?

27. What should the Number of tosses: be set to? Why?

Next, using the applet, carry out 1,000 runs of the simulation and sketch your plot below.

Question:

28. What does each dot on the plot represent?

29. What outcomes, if any, would you consider to be surprising or unusual to observe if female fish really have no preference?

30. In the actual study, all six of the female swordtail fish studied picked the symmetric male swordtail fish. Do you think this result provides evidence that in general, female swordtail fish have a preference for symmetric male swordtail fish? Why or why not?

9

At this point we have encountered several basic examples that have required us to think statistically in order to answer a question of interest. Before we move on to some slightly more complex examples, we will discuss some basic definitions that will be used throughout the semester.

What is Statistics?

Statistics is the _______________ of learning from data.

This involves ______________, _______________, and drawing conclusions from data.

The application of statistics can be divided into _____ broad areas:

Descriptive Statistics : ______________ methods or _____________ summaries which describe data.

Inferential Statistics : The process in which a smaller group (_____________) is used to draw conclusions

about a larger group (________________).

Types of DataRecall that statistics is the science of data. All data can be classified into one of two types of data.

Categorical (Qualitative) Data : Measurements are _________________ into one of a group of categories.

o Nominal

Examples:

o Ordinal

Examples:

Numeric (Quantitative) Data : Measurements that are recorded on a _____________ occurring numeric scale.

o Discrete

Examples:

o Continuous

Examples:

10

Example: National Health SurveyThe National Center for Health Statistics administers a survey to a National Health and Nutrition Examination Survey participants on an annual basis. They survey participants are randomly selected from the U.S. population (http://www.cdc.gov/nchs/nhanes/participant.htm). The following survey items refer to the participant’s dermatological health. Classify the data type that each survey question will yield as nominal, ordinal, discrete, or continuous.

Survey Question Categorical or Numeric? Data Type

How many moles do you have that are at least ¼ inch in diameter?

What is your natural hair color?

When you go outside on a very sunny day for more than an hour, how often do you wear sunscreen (always, most of the time, sometimes, rarely, or never)?

How many times in the past year have you been sunburn?

If applicable, diameter of moles of lesions suspicious of melanoma or other malignancies?

Some Basic DefinitionsMost of what we’ll be doing in this course centers on trying to understand a set of information. This set of information is from a …

Population: The complete collection of _____ elements to be studied.

The population is often so big that obtaining information about all its elements is either difficult or impossible. So, we work with a more manageable set of data obtained from a …

Sample: A ___________________ of elements drawn from a population.

Consider the survey mentioned above. Identify the following:

Population of interest:

Sample:

11

Observation: The collection of ___________________ from a particular unit in a population.

Variable: Any measureable __________________ of an observation.

Example: Claims of Numbness After Automobile AccidentA 28-year-old white woman developed pain involving the spine and the left side of her body after an automobile collision. She was actively involved in a personal litigation against the company that owned the other vehicle, and she reported constant pain and numbness in the left arm. To test her claims, researchers touched her left arm with either 1 finger or 2 fingers simultaneously while her eyes were closed. The word “touch” was said simultaneously with the presentation of the tactile stimulus so that the subject knew when to respond. She then had to indicate whether she felt 1 single touch or 2 simultaneous touches (with the double-touch stimulus, the fingertips were always spaced 2 inches apart). The subject received 100 stimuli overall; she was correct on 30 of them. Is there statistical evidence that she is intentionally answering incorrectly?

Questions:

31. Identify the population of interest.

32. Identify the sample.

33. Identify the single categorical variable of interest.

34. How many times did the patient correctly identify the stimuli? What percentage is this? Note that calculating and reporting this sample proportion is an example of descriptive statistics.

35. If the patient truly suffered from numbness in her arm, how many trials would you expect her to correctly identify the stimulus used? Explain.

Note that the observed number of correct answers is fewer than what would be expected if she suffered from numbness and was simply guessing. A key question is to determine whether this result is surprising under the assumption of numbness. To answer this we will again use the applet to simulate the process of receiving 100 stimuli and indicating which type of stimuli (1 or 2 fingers) was used over and over again. We’ll keep track of how many times the patient was correct (i.e. the number of heads). Once we’ve repeated this process a large number of times, we’ll have a pretty good sense for what outcomes would be very surprising, somewhat surprising or not surprising at all if the patient really suffers from numbness.

12

Set-up the applet to mimic this scenario and carry out 1,000 runs of the simulation. Sketch your plot on the graph below.

Questions:

36. What does each dot on the plot represent?

37. Based on the results of the simulation study, would you consider the result actually obtained in the study (30 out of 100) to be surprising or unusual if the patient really suffers from numbness?

38. Do you think there is statistical evidence the patient is intentionally answering incorrectly? Explain why or why not. Also, note that using the data obtained from the sample to make a generalization about patients who suffer from numbness is an example of inferential statistics.

13

Example: Helper vs. HindererIn a study reported in the November 2007 issue of Nature, researchers investigated whether infants take into account an individual’s actions towards others in evaluating that individual as appealing or aversive, perhaps laying the foundation for social interaction. (Hamlin, Wynn, and Bloom, 2007). In one component of the study, sixteen 10-month old infants were shown a “climber” character (a piece of wood with “google” eyes glued onto it) that could not make it up a hill in two tries. Then they were shown two scenarios for the climber’s next try, one where the climber was pushed to the top of the hill by another character (“helper”) and one where the climber was pushed back down the hill by another character (“hinderer”). The infant was alternately shown these two scenarios several times. Then the child was presented with both pieces of wood (the helper and the hinderer) and was asked to pick one to play with. The color and shape and order (left/right) of the toys was varied and balanced among the 16 infants. A video of the experiment can be found at the following link: http://campuspress.yale.edu/infantlab/media/.

Sources: Introducing Concepts of Statistical Inference. Rossman, Chance, Cobb, and Holcomb. NSF/DUE/CCLI # 0633349 Hamlin, J. Kiley, Karen Wynn, and Paul Bloom. “Social evaluation by preverbal infants.” November 22, 2007. Nature,

Volume 150.

Research Question – Do 10-month old infants tend to prefer the helper toy over the hinderer toy?

Questions:

37. Why is it important for the researchers to balance the color, shape and order of the toys across the study? For example, how would the study results have been affected if the researchers always made the helper toy a blue circle and the hinderer toy a yellow triangle?

38. Identify the population of interest in the study.

39. Identify the sample in the study.

40. What is the single categorical variable of interest in the study?

41. Recall that the study involves 16 10-month old infants. If the population of all 10-month old infants has no real preference for one toy over the other, how many infants would you expect to choose the helper toy? Explain.

42. Suppose that 10 out of the 16 infants chose the helper toy (62.5%). Since this value is more than what is expected (50%), a researcher argues that these data show that the majority of ALL 10-month old infants would choose the helper toy. What is wrong with this reasoning?

14

Once again, the key question is how to determine whether the study’s result is surprising under the assumption that there is no real preference for one toy over the other in the population of all 10-month-old infants. To answer this, we will simulate the process of 16 infants simply choosing a toy at random, over and over again. Each time we simulate the process, we’ll keep track of how many infants out of the 16 chose the helper toy (denoted as heads in the applet). Once we’ve repeated this process a large number of times, we’ll have a pretty good sense for what outcomes would be very surprising, or somewhat surprising, or not so surprising if the population of all 10-month-old infants has no real preference.

In the previous examples, we have used a logical process to make statistical inferences in problems involving a single categorical variable. Next, we will add more structure to these statistical investigations by introducing a procedure known as _______________________________________. Before we discuss this procedure, we need a few more definitions.

Parameter – A numerical descriptive measure of a ___________________. This value is almost always unknown, and our goal is to estimate this parameter or test claims regarding it.

Statistic – A numerical descriptive measure of a ___________________. This value is calculated from the observed data.

Question:

43. Identify the parameter of interest for this study.

44. In the actual study, 14 out of 16 infants chose the helper toy. Identify the statistic for this study.

Hypothesis testing is a procedure, based on sample evidence and probability, used to test a claim regarding a population parameter. The test will measure how well our observed sample statistic agrees with some assumption about this population parameter.

Before you begin a hypothesis test, you should clearly state your research question.

Step 0: Define the research question.

15

Is there evidence that 10-month old infants prefer the Helper toy over the Hinderer toy?

Step 1: Setting up the null and alternative hypotheses.

The null hypothesis, Ho, is what we will assume to be true, and we will evaluate the observed data from our study against what we expected to see under the null hypothesis. This will always contain a statement saying that the population parameter is equal to some value.

The alternative hypothesis, Ha, is what we are trying to show. Therefore, the research question is restated here in the alternative hypothesis. This will always contain statements of inequality, saying that the population parameter is less than, greater than, or different from the value in the null hypothesis.

H0: There is no preference for one toy over the other in the population of all 10-month-olds.

Ha: The majority of all 10-month-old infants prefer the helper toy.

We can also state these hypotheses in terms of the parameter of interest, p (defined in Question 43):

H0:

Ha:

Step 2: Finding the p-value.In the previous examples, we assumed the null hypothesis was true when setting the Probability of heads: in the applet. Then, we used the results simulated under this scenario to help us decide whether observing results such as our sample data would be an unusual event if the null hypothesis were true.

Up to this point, whether an observed result was considered unusual (or extreme) has been a rather subjective decision. Now, we will discuss the guidelines used by statisticians to determine whether an observed result is extreme enough under the null hypothesis to conclude that the evidence supports the research question.

Statisticians use what is called a _________________ to quantify the amount of evidence that an observed result from a set of data provides for a research question.

P-value – The probability of observing an outcome as __________________ (or more extreme) than the observed study result, assuming the _____ hypothesis is true.

Use the instructions outlined previously to carry out the simulation using the applet. Note that you will have to revise a few elements of the simulation that relate to the following questions:

16

What is the probability that each outcome occurs, given that the population of all 10-month-old infants has no real preference for either toy? Change the Probability of heads: accordingly.

How many infants were used in this study? Keep this value in mind when setting the Number of tosses:.

Carry out the simulation study 1,000 times overall, keeping track of the number of infants that choose the helper toy in each trial of the simulation. Sketch in your results below:

Questions:

45. How many infants in the study picked the Helper toy?

46. What additional outcomes would convince you even more that infants prefer the Helper toy over the Hinderer toy? These outcomes are considered to be extreme outcomes.

47. The p-value is the probability of observing an outcome as extreme as observed in the study. Using your simulation results from the applet, find the estimated p-value for this study. You’ll want to enter 14 as

shown: . The p-value will be provided automatically.

Step 3: Making a conclusion in context of the research question.

17

If the p-value is less than ________, then the data provide enough ___________________ evidence to support the research question.

If the p-value is ________ less than 0.05, then the data _____________ provide enough statistical evidence to support the research question.

The conclusion is written in the context of the problem and a person with no statistical background should be able to understand these conclusions (i.e., a conclusion should NOT say something like “We reject the null hypothesis.”)

Question:

48. What is the conclusion for the Helper vs. Hinderer study?

Example: Effectiveness of an Experimental DrugSuppose a commonly prescribed drug for relieving nervous tension is believed to be only 70% effective. Experimental results with a new drug administered to a random sample of 20 adults who were suffering from nervous tension show that 18 received relief.

Research Question - Is there statistical evidence that the new experimental drug is more than 70% effective?

Questions:

49. Identify the population of interest.

50. Identify the parameter of interest.

51. Identify the sample.

52. Identify the statistic of interest.

53. What is the single categorical variable of interest in the study?

18

Step 0: Define the research question.

Is there statistical evidence that the new experimental drug is more than 70% effective?

Step 1: Determine the null and alternative hypotheses.

H0:

Ha:

Step 2: Find the p-value.Use the applet to carry out the simulation for this example. Sketch your plot below.

Questions:

54. What does each dot on the plot represent?

55. How many adults experienced relief?

56. What additional outcomes would convince you even more that the new drug was more effective than the old drug?

57. The p-value is the probability of observing an outcome as extreme as observed in the study. Using your simulation results, find the estimated p-value for this study.

Step 3: Making a conclusion in context of the research question.

19