Validation of the rhinitis outcomes questionnaire (ROQ)

  • Published on

  • View

  • Download

Embed Size (px)


<ul><li><p>Validation of the rhinitis outcomes questionnaire(ROQ)John Santilli, MD*; Robert Nathan, MD; Jeffrey Glassheim, DO; William Rockwell, MD*; andKaren Gold, PhD</p><p>Background: Currently, there is no easy-to-use and comprehensive questionnairethat measures the effectiveness of treatment of rhinitis-related symptoms in a privatepractice setting.Objective: To validate a brief, effective, self-administered rhinitis symptoms</p><p>questionnaire that can easily be used in a private practice setting for trackingtreatment outcomes.Methods: One hundred seventy-five patients were randomly chosen from three</p><p>private practices nationwide and were surveyed regarding systemic problems, nasal,eye, and chest symptoms, as well as medical treatment obtained. This survey wasadministered twice in one sitting with the first administration a recall of symptomsbefore immunotherapy, and the second administration an evaluation of currentsymptoms. Appropriate statistical methods were used to evaluate the validity,reliability, and responsiveness of the questionnaire presented. The validation of thisinstrument included content, statistical, construct, and predictive validity.Results: Both the content and statistical validity were very good in that the</p><p>questionnaire was easily understood and covered required domains. Constructvalidity revealed that the items were sensibly related to the domains that contained themand had less association with items from different domains. The reliability of the scaleswas very good (Cronbachs 0.80) to outstanding (Cronbachs 0.92).Conclusions: Functionality was established for the rhinitis outcomes question-</p><p>naire (ROQ) for use in a private practice setting. This allows private practiceallergists to collect reliable data with the ROQ.</p><p>Ann Allergy Asthma Immunol 2001;86:222225.</p><p>INTRODUCTIONA recent study estimates that up to30% of the population (approximately79.5 million people) may be affectedby allergic rhinitis. It accounts for 3.5million lost workdays and 2 millionmissed school days each year, with anindirect cost reaching $4 billion. Thedirect and indirect costs of this diseaseare over $8.5 billion per year.1,2Currently, there are no validated</p><p>questionnaires for use in a private</p><p>practice setting to evaluate outcomesof the treatment of allergic rhinitis.The best way to track patient progressis capturing current and historical datain one sitting, using a survey that iseasy to understand and short enoughthat the patient fully completes it. Ide-ally, it would be filled out during themandatory wait after an injection. Theuncertainty of such a survey, however,is whether the data collected in thismanner would be statistically mean-ingful.The questionnaire for the study was</p><p>designed by the members of the Rhi-nitis Committee, and revised to includeinput from the Executive Committee ofthe American College of Asthma, Al-lergy, and Immunology. It is designedto be quickly self-administered andmeasure the effectiveness of immuno-therapy based on outcomes such as</p><p>quality of life and symptoms of thedisease. This study evaluates the sta-tistical validity of the rhinitis outcomesquestionnaire (ROQ) relying on a val-idation process that has been used suc-cessfully on different instruments.3</p><p>METHODSPatientsThe questionnaire was distributed to175 randomly selected patients whohad received the diagnosis of allergicrhinitis. The patients were receivingimmunotherapy at one of three privateallergy practices, located in Bridge-port, Connecticut; Colorado Springs,Colorado; and Fresno, California.The pool consisted of 119 female</p><p>patients and 56 male patients, with anaverage age of 38 years. The self-ad-ministered questionnaire was given toeach patient twice in the same sitting,with the first instance a history ofsymptoms before beginning immuno-therapy, and the second an evaluationof current symptoms. The minimumduration of the patients immunother-apy treatment was 12 months, with anaverage duration of 3.3 years. Immu-notherapy treatment was, for any onegroup, or combination of, pollens,molds, mites, and animal dander.Questionnaire DesignThe ROQ was adapted from a previousquestionnaire4 by the members of theRhinitis Committee of the AmericanCollege of Allergy, Asthma, and Im-munology, and modified to include in-put from the Colleges Executive Com-mittee. The questionnaire was designedto be completed in 5 minutes or less.The symptoms questionnaire is di-</p><p>vided into five parts. The first partevaluates global symptoms, and thesecond, third, and fourth parts com-prise specific symptoms of the nose,</p><p>* Private practice, Bridgeport, Connecticut. Private practice, Colorado Springs, Colorado. Private practice, Fresno, California. Georgetown University.The American College of Allergy, Asthma,</p><p>and Immunology funded the statistical validationfor this study.Received for publication April 12, 1999.Accepted for publication in revised form June</p><p>20, 2000.</p><p>222 ANNALS OF ALLERGY, ASTHMA, &amp; IMMUNOLOGY</p></li><li><p>eyes, and chest. Each symptom ques-tion was scored on a Likert scale from0 to 5 indicating the level of involve-ment (0 no; 1 minimal; 2 mild;3 moderate; 4 severe; and 5 very severe). The fifth component ofthe evaluation collected a symptom-specific medical treatment history byfrequency counts, including hospital-ization, emergency room visits, sinussurgery, antibiotics, and other medica-tion use.Statistical AnalysisThis study uses relevant statisticalmethods of assessing the validity, reli-ability, and responsiveness of theROQ, relying on a validation processthat has been used successfully on dif-ferent instruments. Four types of valid-ity were used as criteria for evaluatingthe quality of this instrument: 1) con-tent validity; 2) statistical validity; 3)construct validity; and 4) predictivevalidity. Additionally, the reliability ofthe scale used to score symptom re-sponses and the overall responsivenessof the questionnaire were evaluated.Content validity is a substantive</p><p>evaluation of the adequacy of the itemsincluded in the instrument to measurethe domains of interest. To evaluatethe content validity of the question-naire, allergists recruited from the Ex-ecutive Committee of the AmericanCollege of Asthma, Allergy, and Im-munology substantively evaluated theinstrument for completeness, clarity,and ease of use.Statistical validity is the numeric</p><p>quality and characteristics of the dataso as to permit sensible statistical anal-ysis. For example, if all subjects pro-vide an answer of zero to one of theitems in a scale, this does not havevariability and would not be amenableto further statistical analysis. Descrip-tive statistics and histograms were gen-erated for each item on the question-naire to evaluate statistical validity.Construct validity is determination</p><p>of the interrelationships between theitems on an instrument to verify thatscores or the substantive grouping ofitems is empirically justified. Con-struct validity was evaluated by exam-</p><p>ining an item level Pearson correlationmatrix on the first collection of symp-tom measures. Pearson correlation is ameasure of association between twovariables. The values of the correlationrun between 1 and 1; 1 indicates aperfect reverse association, 0 indicatesno association, and 1 indicates a per-fect association. The Pearson correla-tion matrix consists of the Pearson cor-relation of all pairs of items containedin the survey.The criteria used were adapted from</p><p>the standard multi-trait, multi-methodanalysis. Essentially, if the system/or-gan-oriented structure of this question-naire is to be substantiated, eye itemsshould be more highly correlated witheye items than with ear items. Also, ifit is believed that symptom types arerelated across organs, then irritationitems should be more highly correlatedwith irritation items than with dis-charge items. These two criteria ofhigh correlation between like itemsand lower correlation between differ-ent items are called the convergent anddivergent criteria, respectively.Predictive validity determines the</p><p>predictive power of symptoms to fu-ture events. Certain symptoms are as-sociated with medical treatments, andthis informs us of the predictive valid-ity of the items and scores for trackingmedical resource usage. Symptoms as-sociated with costly medical resources,such as emergency room visits, sur-gery, and hospitalization, are exam-ined. The present study is designed todemonstrate the predictive validity ofthe items and scores for emergencyroom visits, sinus surgery, and hospi-talization.The reliability of the scale used was</p><p>also evaluated, and can be understoodin terms of a signal to noise ratio.Cronbachs , a standard measure ofinternal consistency, was used for theevaluation of the reliability of thequestionnaire on the first administra-tion of the symptom measures. Reli-ability ranges in value from 0 (no in-formation and all noise) to 1 (allinformation and no noise). Generally,scales with less than 0.6 reliability areconsidered inferior, 0.8 are research</p><p>quality, and 0.9 or better superior interms of reliability.To evaluate the responsiveness of</p><p>the instrument, the responses of thepatients were analyzed. Because thesedata were collected to measure effective-ness of immunotherapy, a responsiveinstrument would be expected to showdifferences in the degree of symptomsbefore immunotherapy compared withafter receiving immunotherapy.</p><p>RESULTSContent ValidityThe questionnaire was evaluated by al-lergists recruited from the ExecutiveCommittee of the American College ofAsthma, Allergy, and Immunology,and their input was included in thequestionnaire design.Statistical ValidityThe items generally showed excellentvariability from the descriptive statis-tics and histograms that were gener-ated for each item on the question-naire. Frequency distributions wereexamined for all items. All items wereunimodal, although several items hadsizably skewed distributions, indicat-ing that many subjects in the samplenever or rarely suffered from particularsymptoms. This was most pronouncedwith respect to the fatigue item, ofwhich 26.1% indicated no symptom,9% indicated a mild problem, and23.1% indicated a moderate problem.Skewed distributions were greatly di-minished when the aggregate scoreswere examined. Means and standarddeviations are reported in Table 1 andTable 2.Construct ValidityThe Pearson correlations were as ex-pected for this type of instrument. Aninitial grouping of items according totheir basis (nose, eye, chest, etc.) wasconstructed. The average intercorrela-tion of the nasal symptoms was rela-tively high (r 0.3 to 0.4, P .0001),and the average intercorrelation amongeye symptoms was high (r 0.7, P .0001), as were the chest and breathingsymptoms (r 0.5 to 0.6, P .0001).Also, fatigue, sleep problems, muscle</p><p>VOLUME 86, FEBRUARY, 2001 223</p></li><li><p>weakness, irritability, and dizzinesswere, for the most part, sizably inter-correlated (r 0.4, P .0001). Sinus,postnasal drip, and sore throat weresizably intercorrelated, presumably at-tributable to issues of cause and effect(ie, postnasal drip causing throat irri-tation) with r 0.5, P .0001.Further interrelations among groups</p><p>of items were established based on phys-ical proximity and high intercorrelation.Nasal headache/pressure, postnasal drip,</p><p>sinus pain, snoring, sneezing, and sorethroat warranted grouping (r 0.4 to0.5, P .0001), with marginal associa-tion of the ear (r 0.3, P .0003). Thisreconfirms the classic ear, nose, andthroat associations. The set of highly in-tercorrelated eye items (itching, redness,watering, and burning) had sizable cor-relations with sneezing, mucus, andsmell, but these additional correlationswere of approximately half the magni-tude (r 0.4, P .0001) of the eye-</p><p>only item intercorrelations. Finally, thechest pain item was sizably correlatedwith the other items (r 0.35, P .0005). This is most likely anothercause-and-effect association of chestpain, creating a sense of muscle weak-ness and sleep problems.The above correlations are consis-</p><p>tent with expectations and confirmsensible convergence among like or bi-ologically associated items. All otherassociations of items have trivial cor-</p><p>Table 1. Descriptive Statistics &amp; Responsiveness by Scale</p><p>Scale VariableBaselineMean</p><p>StandardDeviation</p><p>MinimumObserved</p><p>MaximumObserved</p><p>AverageDecrease</p><p>PercentageDecrease</p><p>Nose 24.87 10.4 0 50 14.38 58%Eye 8.54 5.5 0 20 4.97 58%Chest 10.82 7.6 0 30 5.99 55%System 7.29 5.5 0 20 3.24 44%</p><p>Table 2. Descriptive Statistics &amp; Responsiveness by Item</p><p>Item VariableBaselineMean</p><p>StandardDeviation</p><p>MinimumObserved</p><p>MaximumObserved</p><p>AverageDecrease</p><p>PercentageDecrease</p><p>Smell 1.41 1.51 0 5 0.84 60%Eye Watering 2.16 1.55 0 5 1.29 60%Wheezing 1.87 1.70 0 5 1.13 60%Sore Throat 2.22 1.41 0 5 1.31 59%Nasal Itching 2.37 1.53 0 5 1.39 59%Eye Burning 1.77 1.56 0 5 1.04 59%Nasal Green Mucus 2.31 1.54 0 5 1.34 58%Earache 1.20 1.59 0 5 0.69 58%Eye Redness 2.13 1.52 0 5 1.19 56%Chest Pain 0.93 1.35 0 5 0.52 56%Coughing 2.16 1.50 0 5 1.19 55%Chest Tightness 1.75 1.67 0 5 0.96 55%Phlegm 2.06 1.64 0 5 1.13 55%Nasal Blockage 3.19 1.41 0 5 1.67 52%Watery Mucus 2.45 1.50 0 5 1.25 51%Eye Itching 2.48 1.48 0 5 1.27 51%Breath 2.04 1.75 0 5 1.05 51%Post Nasal Drip 2.93 1.45 0 5 1.43 49%Sinus Pain 2.81 1.63 0 5 1.37 49%Sleeping 1.86 1.65 0 5 0.89 48%Irritability 1.76 1.54 0 5 0.85 48%Dizziness 0.85 1.31 0 5 0.41 48%Sneezing 2.65 1.40 0 5 1.27 48%Muscle 0.80 1.22 0 4 0.35 44%Snoring 1.30 1.62 0 5 0.54 42%Fatigue 2.06 1.66 0 5 0.79 38%</p><p>Nathan RA, Meltner EO, Selner JC, Storms W. Prevalence of Allergic Rhinitis in the United States. J Allergy Clin Immunol 1997;99:S808S814.Adkinson NF. Immunotherapy for Allergic Rhinitis. N Engl J Med 1999;341:522.Wasserfallen JB, Gold K, Schulman KA, Baraniuk JN. Development and validation of a rhinitis and asthma symptom score for use as an outcomemeasure in clinical trails. J Allergy Clin Immunol 1997;100:1622.Storms WW, Nathan RA, Bodman SF, Byer P. Improving the treatment of nocturnal asthma: use of an office questionnaire to identify nocturnalasthma symptoms. J Asthma 1996;33:1658.</p><p>224 ANNALS OF ALLERGY, ASTHMA, &amp; IMMUNOLOGY</p></li><li><p>relations that average approximatelyr 0.15 and are almost uniformly notsignificant. This confirms a sensibledivergence among unlike items.Predictive ValidityAt the item level, severity of wheezing,chest pain, and chest tightness is sig-nificantly correlated with emergencyroom visits. (These correlations arevalued between 0.3 and 0.4.) Further,severity of earaches, shortness ofbreath, and coughing are also associ-ated with emergency room visits withmarginally significant correlations ofapproximately 0.2. Chest pain is mar-ginally significantly associated withhospitalization, with a correlation of0.25. All correlations presented hereare significant (P .05). Additionally,the severity of earaches, shortness ofbreath, wheezing, chest tightness, andpain are significantly associated withloss of workdays. (These correlationsrun from approximately 0.3 to 0.4.)ReliabilityCronbachs was used for the evalu-ation of the reliabili...</p></li></ul>


View more >