Validation of the rhinitis outcomes questionnaire(ROQ)John Santilli, MD*; Robert Nathan, MD†; Jeffrey Glassheim, DO‡; William Rockwell, MD*; andKaren Gold, PhD§

Background: Currently, there is no easy-to-use and comprehensive questionnairethat measures the effectiveness of treatment of rhinitis-related symptoms in a privatepractice setting.Objective: To validate a brief, effective, self-administered rhinitis symptoms

questionnaire that can easily be used in a private practice setting for trackingtreatment outcomes.Methods: One hundred seventy-five patients were randomly chosen from three

private practices nationwide and were surveyed regarding systemic problems, nasal,eye, and chest symptoms, as well as medical treatment obtained. This survey wasadministered twice in one sitting with the first administration a recall of symptomsbefore immunotherapy, and the second administration an evaluation of currentsymptoms. Appropriate statistical methods were used to evaluate the validity,reliability, and responsiveness of the questionnaire presented. The validation of thisinstrument included content, statistical, construct, and predictive validity.Results: Both the content and statistical validity were very good in that the

questionnaire was easily understood and covered required domains. Constructvalidity revealed that the items were sensibly related to the domains that contained themand had less association with items from different domains. The reliability of the scaleswas very good (Cronbach’s � � 0.80) to outstanding (Cronbach’s � � 0.92).Conclusions: Functionality was established for the rhinitis outcomes question-

naire (ROQ) for use in a private practice setting. This allows private practiceallergists to collect reliable data with the ROQ.

Ann Allergy Asthma Immunol 2001;86:222–225.

INTRODUCTIONA recent study estimates that up to30% of the population (approximately79.5 million people) may be affectedby allergic rhinitis. It accounts for 3.5million lost workdays and 2 millionmissed school days each year, with anindirect cost reaching $4 billion. Thedirect and indirect costs of this diseaseare over $8.5 billion per year.1,2Currently, there are no validated

questionnaires for use in a private

practice setting to evaluate outcomesof the treatment of allergic rhinitis.The best way to track patient progressis capturing current and historical datain one sitting, using a survey that iseasy to understand and short enoughthat the patient fully completes it. Ide-ally, it would be filled out during themandatory wait after an injection. Theuncertainty of such a survey, however,is whether the data collected in thismanner would be statistically mean-ingful.The questionnaire for the study was

designed by the members of the Rhi-nitis Committee, and revised to includeinput from the Executive Committee ofthe American College of Asthma, Al-lergy, and Immunology. It is designedto be quickly self-administered andmeasure the effectiveness of immuno-therapy based on outcomes such as

quality of life and symptoms of thedisease. This study evaluates the sta-tistical validity of the rhinitis outcomesquestionnaire (ROQ) relying on a val-idation process that has been used suc-cessfully on different instruments.3

METHODSPatientsThe questionnaire was distributed to175 randomly selected patients whohad received the diagnosis of allergicrhinitis. The patients were receivingimmunotherapy at one of three privateallergy practices, located in Bridge-port, Connecticut; Colorado Springs,Colorado; and Fresno, California.The pool consisted of 119 female

patients and 56 male patients, with anaverage age of 38 years. The self-ad-ministered questionnaire was given toeach patient twice in the same sitting,with the first instance a history ofsymptoms before beginning immuno-therapy, and the second an evaluationof current symptoms. The minimumduration of the patient’s immunother-apy treatment was 12 months, with anaverage duration of 3.3 years. Immu-notherapy treatment was, for any onegroup, or combination of, pollens,molds, mites, and animal dander.Questionnaire DesignThe ROQ was adapted from a previousquestionnaire4 by the members of theRhinitis Committee of the AmericanCollege of Allergy, Asthma, and Im-munology, and modified to include in-put from the College’s Executive Com-mittee. The questionnaire was designedto be completed in 5 minutes or less.The symptoms questionnaire is di-

vided into five parts. The first partevaluates global symptoms, and thesecond, third, and fourth parts com-prise specific symptoms of the nose,

* Private practice, Bridgeport, Connecticut.† Private practice, Colorado Springs, Colorado.‡ Private practice, Fresno, California.§ Georgetown University.The American College of Allergy, Asthma,

and Immunology funded the statistical validationfor this study.Received for publication April 12, 1999.Accepted for publication in revised form June

20, 2000.

222 ANNALS OF ALLERGY, ASTHMA, & IMMUNOLOGY

eyes, and chest. Each symptom ques-tion was scored on a Likert scale from0 to 5 indicating the level of involve-ment (0 � no; 1 � minimal; 2 � mild;3 � moderate; 4 � severe; and 5 �very severe). The fifth component ofthe evaluation collected a symptom-specific medical treatment history byfrequency counts, including hospital-ization, emergency room visits, sinussurgery, antibiotics, and other medica-tion use.Statistical AnalysisThis study uses relevant statisticalmethods of assessing the validity, reli-ability, and responsiveness of theROQ, relying on a validation processthat has been used successfully on dif-ferent instruments. Four types of valid-ity were used as criteria for evaluatingthe quality of this instrument: 1) con-tent validity; 2) statistical validity; 3)construct validity; and 4) predictivevalidity. Additionally, the reliability ofthe scale used to score symptom re-sponses and the overall responsivenessof the questionnaire were evaluated.Content validity is a substantive

evaluation of the adequacy of the itemsincluded in the instrument to measurethe domains of interest. To evaluatethe content validity of the question-naire, allergists recruited from the Ex-ecutive Committee of the AmericanCollege of Asthma, Allergy, and Im-munology substantively evaluated theinstrument for completeness, clarity,and ease of use.Statistical validity is the numeric

quality and characteristics of the dataso as to permit sensible statistical anal-ysis. For example, if all subjects pro-vide an answer of zero to one of theitems in a scale, this does not havevariability and would not be amenableto further statistical analysis. Descrip-tive statistics and histograms were gen-erated for each item on the question-naire to evaluate statistical validity.Construct validity is determination

of the interrelationships between theitems on an instrument to verify thatscores or the substantive grouping ofitems is empirically justified. Con-struct validity was evaluated by exam-

ining an item level Pearson correlationmatrix on the first collection of symp-tom measures. Pearson correlation is ameasure of association between twovariables. The values of the correlationrun between �1 and 1; �1 indicates aperfect reverse association, 0 indicatesno association, and 1 indicates a per-fect association. The Pearson correla-tion matrix consists of the Pearson cor-relation of all pairs of items containedin the survey.The criteria used were adapted from

the standard multi-trait, multi-methodanalysis. Essentially, if the system/or-gan-oriented structure of this question-naire is to be substantiated, eye itemsshould be more highly correlated witheye items than with ear items. Also, ifit is believed that symptom types arerelated across organs, then irritationitems should be more highly correlatedwith irritation items than with dis-charge items. These two criteria ofhigh correlation between like itemsand lower correlation between differ-ent items are called the convergent anddivergent criteria, respectively.Predictive validity determines the

predictive power of symptoms to fu-ture events. Certain symptoms are as-sociated with medical treatments, andthis informs us of the predictive valid-ity of the items and scores for trackingmedical resource usage. Symptoms as-sociated with costly medical resources,such as emergency room visits, sur-gery, and hospitalization, are exam-ined. The present study is designed todemonstrate the predictive validity ofthe items and scores for emergencyroom visits, sinus surgery, and hospi-talization.The reliability of the scale used was

also evaluated, and can be understoodin terms of a signal to noise ratio.Cronbach’s �, a standard measure ofinternal consistency, was used for theevaluation of the reliability of thequestionnaire on the first administra-tion of the symptom measures. Reli-ability ranges in value from 0 (no in-formation and all noise) to 1 (allinformation and no noise). Generally,scales with less than 0.6 reliability areconsidered inferior, 0.8 are research

quality, and 0.9 or better superior interms of reliability.To evaluate the responsiveness of

the instrument, the responses of thepatients were analyzed. Because thesedata were collected to measure effective-ness of immunotherapy, a responsiveinstrument would be expected to showdifferences in the degree of symptomsbefore immunotherapy compared withafter receiving immunotherapy.

RESULTSContent ValidityThe questionnaire was evaluated by al-lergists recruited from the ExecutiveCommittee of the American College ofAsthma, Allergy, and Immunology,and their input was included in thequestionnaire design.Statistical ValidityThe items generally showed excellentvariability from the descriptive statis-tics and histograms that were gener-ated for each item on the question-naire. Frequency distributions wereexamined for all items. All items wereunimodal, although several items hadsizably skewed distributions, indicat-ing that many subjects in the samplenever or rarely suffered from particularsymptoms. This was most pronouncedwith respect to the fatigue item, ofwhich 26.1% indicated no symptom,9% indicated a mild problem, and23.1% indicated a moderate problem.Skewed distributions were greatly di-minished when the aggregate scoreswere examined. Means and standarddeviations are reported in Table 1 andTable 2.Construct ValidityThe Pearson correlations were as ex-pected for this type of instrument. Aninitial grouping of items according totheir basis (nose, eye, chest, etc.) wasconstructed. The average intercorrela-tion of the nasal symptoms was rela-tively high (r� 0.3 to 0.4, P� .0001),and the average intercorrelation amongeye symptoms was high (r � 0.7, P �.0001), as were the chest and breathingsymptoms (r � 0.5 to 0.6, P � .0001).Also, fatigue, sleep problems, muscle

VOLUME 86, FEBRUARY, 2001 223

weakness, irritability, and dizzinesswere, for the most part, sizably inter-correlated (r � 0.4, P � .0001). Sinus,postnasal drip, and sore throat weresizably intercorrelated, presumably at-tributable to issues of cause and effect(ie, postnasal drip causing throat irri-tation) with r � 0.5, P � .0001.Further interrelations among groups

of items were established based on phys-ical proximity and high intercorrelation.Nasal headache/pressure, postnasal drip,

sinus pain, snoring, sneezing, and sorethroat warranted grouping (r � 0.4 to0.5, P � .0001), with marginal associa-tion of the ear (r� 0.3, P� .0003). Thisreconfirms the classic ear, nose, andthroat associations. The set of highly in-tercorrelated eye items (itching, redness,watering, and burning) had sizable cor-relations with sneezing, mucus, andsmell, but these additional correlationswere of approximately half the magni-tude (r � 0.4, P � .0001) of the eye-

only item intercorrelations. Finally, thechest pain item was sizably correlatedwith the other items (r � 0.35, P �.0005). This is most likely anothercause-and-effect association of chestpain, creating a sense of muscle weak-ness and sleep problems.The above correlations are consis-

tent with expectations and confirmsensible convergence among like or bi-ologically associated items. All otherassociations of items have trivial cor-

Table 1. Descriptive Statistics & Responsiveness by Scale

Scale VariableBaseline

MeanStandardDeviation

MinimumObserved

MaximumObserved

AverageDecrease

PercentageDecrease

Nose 24.87 10.4 0 50 14.38 58%Eye 8.54 5.5 0 20 4.97 58%Chest 10.82 7.6 0 30 5.99 55%System 7.29 5.5 0 20 3.24 44%

Table 2. Descriptive Statistics & Responsiveness by Item

Item VariableBaseline

MeanStandardDeviation

MinimumObserved

MaximumObserved

AverageDecrease

PercentageDecrease

Smell 1.41 1.51 0 5 0.84 60%Eye Watering 2.16 1.55 0 5 1.29 60%Wheezing 1.87 1.70 0 5 1.13 60%Sore Throat 2.22 1.41 0 5 1.31 59%Nasal Itching 2.37 1.53 0 5 1.39 59%Eye Burning 1.77 1.56 0 5 1.04 59%Nasal Green Mucus 2.31 1.54 0 5 1.34 58%Earache 1.20 1.59 0 5 0.69 58%Eye Redness 2.13 1.52 0 5 1.19 56%Chest Pain 0.93 1.35 0 5 0.52 56%Coughing 2.16 1.50 0 5 1.19 55%Chest Tightness 1.75 1.67 0 5 0.96 55%Phlegm 2.06 1.64 0 5 1.13 55%Nasal Blockage 3.19 1.41 0 5 1.67 52%Watery Mucus 2.45 1.50 0 5 1.25 51%Eye Itching 2.48 1.48 0 5 1.27 51%Breath 2.04 1.75 0 5 1.05 51%Post Nasal Drip 2.93 1.45 0 5 1.43 49%Sinus Pain 2.81 1.63 0 5 1.37 49%Sleeping 1.86 1.65 0 5 0.89 48%Irritability 1.76 1.54 0 5 0.85 48%Dizziness 0.85 1.31 0 5 0.41 48%Sneezing 2.65 1.40 0 5 1.27 48%Muscle 0.80 1.22 0 4 0.35 44%Snoring 1.30 1.62 0 5 0.54 42%Fatigue 2.06 1.66 0 5 0.79 38%

Nathan RA, Meltner EO, Selner JC, Storms W. Prevalence of Allergic Rhinitis in the United States. J Allergy Clin Immunol 1997;99:S808–S814.Adkinson NF. Immunotherapy for Allergic Rhinitis. N Engl J Med 1999;341:522.Wasserfallen JB, Gold K, Schulman KA, Baraniuk JN. Development and validation of a rhinitis and asthma symptom score for use as an outcomemeasure in clinical trails. J Allergy Clin Immunol 1997;100:16–22.Storms WW, Nathan RA, Bodman SF, Byer P. Improving the treatment of nocturnal asthma: use of an office questionnaire to identify nocturnalasthma symptoms. J Asthma 1996;33:165–8.

224 ANNALS OF ALLERGY, ASTHMA, & IMMUNOLOGY

relations that average approximatelyr � 0.15 and are almost uniformly notsignificant. This confirms a sensibledivergence among unlike items.Predictive ValidityAt the item level, severity of wheezing,chest pain, and chest tightness is sig-nificantly correlated with emergencyroom visits. (These correlations arevalued between 0.3 and 0.4.) Further,severity of earaches, shortness ofbreath, and coughing are also associ-ated with emergency room visits withmarginally significant correlations ofapproximately 0.2. Chest pain is mar-ginally significantly associated withhospitalization, with a correlation of0.25. All correlations presented hereare significant (P� .05). Additionally,the severity of earaches, shortness ofbreath, wheezing, chest tightness, andpain are significantly associated withloss of workdays. (These correlationsrun from approximately 0.3 to 0.4.)ReliabilityCronbach’s � was used for the evalu-ation of the reliability of the question-naire. The global symptoms were 0.80,nasal symptoms 0.84, eye symptoms0.92, and chest symptoms 0.88. Thisindicates that all the scales on this in-strument are highly reliable for thepopulation of rhinitis sufferers.Responsiveness of the InstrumentThe data were collected to measure theeffectiveness of immunotherapy, andthere is a sizable, statistically signifi-cant decrease in symptoms as mea-sured by this instrument for both theitem level and scale level (P � .0002).There are two striking observations re-garding Tables 1 and 2. First, everysingle item is responsive. This infor-mation, in conjunction with the fact

that none of the pairs of items havecorrelations greater than 0.85, suggeststhat it would be unwise to reduce theitem pool because responsive informa-tion not contained in the remainingitems would be lost. The most respon-sive set of items are those associatedwith the nose, which is not surprisinggiven that the subjects are under treat-ment for rhinitis. The magnitude of thedecrease in any given symptom rangesfrom approximately 0.5 point to 1.5points. This translates to a change insymptoms in the average subject froma mild/moderate problem to only aminimal problem.

DISCUSSIONThis validation study allowed the de-velopment of the ROQ for use in aprivate practice setting. The objectiveof this study is to provide private prac-tice allergists with an inexpensive andeffective tool to collect outcomes ofallergic rhinitis treatment, specificallyimmunotherapy. The statistical valida-tion of this questionnaire allows pri-vate practice allergists to collect reli-able data with the ROQ.This validation study has several

limitations. The first one is the rela-tively small number of patients. How-ever, the results observed were consis-tent when both the whole populationand the different subgroups were ex-amined, suggesting confidence in thevalidity of the conclusions reached.The second limitation is linked to theinclusion criteria. All patients were onan immunotherapy program for a min-imum of 12 months in the retrospectivestudy. Therefore, patients who hadstopped immunotherapy before this forany reason (ie, failure to notice anyimprovement in symptoms, cost,changing health care plans, etc.) were

excluded from the study. This factormay influence the final statistics; how-ever, it can only be addressed correctlyin a prospective study.

CONCLUSIONThis retrospective study served to val-idate the use of the rhinitis outcomesquestionnaire in patients with rhinitisin private practice settings. Evaluationof responsiveness, statistical validity,external and internal reliabilities, andconstruct validity in accordance withpublished guidelines allowed the es-tablishment of an optimal six-pointscale, and the elimination of irrelevantand redundant items. Using appropri-ate statistical methods of assessing thevalidity, reliability, and responsivenessof the instrument presented, its func-tionality has been established.

REFERENCES1. Nathan RA, Meltner, EO, Selner JC, etal. Prevalence of allergic rhinitis in theUnited States. J Allergy Clin Immunol1997;99:S808–814.

2. Adkinson, NF. Immunotherapy for al-lergic rhinitis. N Engl J Med 1999;341:522–524.

3. Wasserfallen JB, Gold K, SchulmanKA, et al. Development and validationof a rhinitis and asthma symptom scorefor use as an outcome measure in clin-ical trails. J Allergy Clin Immunol1997;100:16–22.

4. Storms WW, Nathan RA, Bodman SF,et al. Improving the treatment of noctur-nal asthma: use of an office question-naire to identify nocturnal asthma symp-toms. J Asthma 1996;33:165–168.

Requests for reprints should be addressed to:John Santilli, MD4675 Main StreetBridgeport, CT 06606E-mail: john@santilli.org

VOLUME 86, FEBRUARY, 2001 225