Validation of the Rhinoplasty Outcomes Evaluation (ROE) questionnaire adapted to Brazilian Portuguese

  • Published on

  • View

  • Download

Embed Size (px)


  • Validation of the Rhinoplasty Outcomes Evaluation (ROE)questionnaire adapted to Brazilian Portuguese

    Suemy Cioffi Izu Eduardo Macoto Kosugi Alessandra Stanquini Lopes

    Karen Vitols Brandao Leonardo Bomediano Garcia Sousa Vincius Magalhaes Suguri

    Luis Carlos Gregorio

    Accepted: 18 September 2013 / Published online: 1 October 2013

    Springer Science+Business Media Dordrecht 2013


    Purpose The aim of this study was to validate the Rhi-

    noplasty Outcomes Evaluation (ROE) questionnaire adap-

    ted to the Brazilian Portuguese.

    Method A prospective study was conducted with ROE

    administration to 56 patients submitted to rhinoplasty

    (preoperatively, and then 15-day and 90-day postopera-

    tively) and 100 volunteers without the need or desire of

    cosmetic or functional nasal surgery. Reliability (internal

    consistency and testretest reproducibility), validity,

    responsiveness and clinical interpretability were assessed.

    Results Rhinoplasty patients mean preoperative score

    was 7.14, 15 days post-op 17.73 and 90 days post-op

    20.50, while controls presented 17.94 points (p \ 0.0001),showing the questionnaires validity and responsiveness.

    Internal consistency was 0.86. Inter- and intra-examiner

    testretest reproducibility was 0.90 and 0.94, respectively.

    The effect size caused by the surgery was considered large

    (15 days post-op compared to the preoperative score: effect

    size = 3.22; 90 days post-op compared to preoperative

    score: effect size = 4.06). The minimally important dif-

    ference was 8.67 points, so changes smaller than 9 points in

    ROE might not be perceived by the patient as an

    improvement or worsening.

    Conclusion The Brazilian Portuguese version of ROE is a

    valid instrument to assess results in rhinoplasty patients.

    Keywords Rhinoplasty Quality of life Questionnaires


    Physicians concept of a good surgical outcome could be

    very different from patients thoughts. Most likely, the

    professional experience of surgeons leads to different

    expectations compared to patients [1]. A given surgeon

    may be satisfied with his or her results, but if patients

    themselves are not similarly pleased, then the intervention

    cannot totally be considered a success [2]. However,

    patient satisfaction could be influenced by many variables,

    such as availability and convenience of health care, the

    bedside manner of the doctor, affability of the extended

    team and perceived cleanliness of the hospital, which can

    complicate evaluation of clinical outcome [3, 4].

    In this context, the quality of life questionnaires are very

    suitable tools that allow quantitative assessment of other-

    wise subjective results, such as patient satisfaction and,

    consequently, surgery success [3]. Quality of life can be

    defined as individuals perception of their position in life

    in the context of the culture and value systems in which

    they live and in relation to their goals, expectations, stan-

    dards and concerns. It is a broad ranging concept affected

    in a complex way by the persons physical health, psy-

    chological state, level of independence, social relationships

    and their relationship to salient features of their environ-

    ment [5]. In aesthetic interventions, more than any other

    aspect of rhinology, patient satisfaction and quality of life

    must be the measure against which successful procedure

    should be judged [3].

    Based on such philosophy, Alsarraf developed four new

    outcome instruments to assess the results of cosmetic facial

    procedures: Rhinoplasty Outcomes Evaluation (ROE),

    Facelift Outcomes Evaluation (FOE), Blepharoplasty

    Outcomes Evaluation and Skin Rejuvenation Outcomes

    Evaluation (SROE) [2, 6]. In order to measure outcomes

    S. C. Izu E. M. Kosugi (&) A. S. Lopes K. V. Brandao L. B. G. Sousa V. M. Suguri L. C. GregorioDepartment of Otorhinolaryngology and Head and Neck

    Surgery, UNIFESP-EPM, Sao Paulo, SP, Brazil



    Qual Life Res (2014) 23:953958

    DOI 10.1007/s11136-013-0539-x

  • such as patient satisfaction and quality of life in the facial

    plastic surgery patient, he identified three key aspects that

    constituted such satisfaction for each treatment modality of

    interest: physical, emotional and social factor. Emotional

    and social factors were similar among each of the specific

    interventions assessed by Alsarraf [2], while physical fac-

    tors presented very specific characteristics depending on

    the surgical option.

    ROE was translated and cross-culturally adapted to

    Brazilian Portuguese [7], according to Guillemin et al. [8].

    However, validation process was not carried out. The

    objective of this study is to validate the Brazilian Portu-

    guese version of the ROE.


    This research was approved by the Ethics in Research

    Committee of the institution, under protocol number CEP

    1791/11, and all participants were volunteers and signed

    the informed consent form.

    The Brazilian Portuguese ROE questionnaire [7] is

    shown in Fig. 1. It is comprised of six questions; each of

    the six items was scored on a 04 scale, with 0 repre-

    senting the most negative response and 4 representing the

    most positive response. Therefore, the total score can vary

    from 0 to 24. In order to facilitate the comprehension of

    the results, the total score can be divided by 24 and

    multiplied by 100, so that the score can vary from 0 to

    100 %. So, 24 points or 100 % means the most patient

    satisfaction [2].

    Recruitment of patients

    Volunteer patients that desired cosmetic nasal surgery

    (rhinoplasty) with or without functional surgery (septo-

    plasty and/or turbinectomy) were recruited from a rhinol-

    ogy unit to be part of rhinoplasty group. Exclusion criteria

    comprised the desire not to participate in the study or

    revision cases. Brazilian Portuguese ROE questionnaire

    was filled in by participants in the preoperative (pre-op)

    and in 15 and 90 days postoperative periods (PO15d and

    PO90d, respectively). In the preoperative period, the

    questionnaire was administered by two of the authors

    (A.S.L. and K.V.B.) and was repeated by one of them after

    15 days to check reproducibility. Afterward, the patients

    were assessed by one of the authors (A.S.L. or K.V.B.) in

    the PO15d and in the PO90d. In the postoperative evalu-

    ations, the patients also answered whether they felt: much

    better, a little better, about the same, a little worse or much

    worse, than the period before the intervention.

    The control group was made up of healthy volunteers

    with no desire or need for aesthetic and/or functional nasal

    surgery, who were recruited from the clinic, employees and

    students or relatives from our institution.

    Analysis of data

    Reliability was analyzed in two ways: internal consistency

    and testretest reproducibility. Internal consistency refer-

    red to the way individual items relate to each other, in order

    to provide homogeneity among them, and was measured

    using Cronbachs alpha [9]. The minimum acceptable score

    Fig. 1 Brazilian Portugueseversion of Rhinoplasty

    Outcomes Evaluation


    954 Qual Life Res (2014) 23:953958


  • for Cronbachs alpha is 0.7 [1012]. Testretest repro-

    ducibility measured the stability of an instrument over time

    after repeated testing [10, 11]. The questionnaire was

    applied twice in 15 days by the same examiner (intra-

    examiner testretest reproducibility) and twice in the same

    day by two different examiners (inter-examiner testretest

    reproducibility) [10]. The correlation between the scores,

    measured by intra-class correlation coefficient (ICC), must

    be of, at least, 0.7 [10, 11].

    Measures validity is the capacity of the questionnaire to

    reflect differences among known groups (diseased vs. non-

    diseased) using the unpaired t test or MannWhitney

    U test, depending on samples homogeneity of variance

    measured by Levenes test and normal distribution mea-

    sured by KolmogorovSmirnov test. Basically, Brazilian

    Portuguese ROE should be able to distinguish the group of

    patients that wish/need rhinoplasty from the group of

    subjects that did not [10, 11].

    Responsiveness is the ability of the questionnaire to

    detect clinical differences over time. Pre- and postoperative

    scores were compared using paired t test or Wilcoxon

    signed-rank test, depending on samples homogeneity of

    variance measured by Levenes test and normal distribu-

    tion measured by KolmogorovSmirnov test. Also,

    responsiveness was assessed by measuring the effect size:

    the mean change score divided by baseline standard devi-

    ation. By convention, an effect size of [0.2 is consideredsmall, [0.5 moderate and [0.8 a large improvement inhealth quality of life [10, 11].

    Clinical interpretability can be calculated by the mini-

    mally important difference (MID), the smallest change in

    scores that a group of patients can detect as a real

    improvement [13]. In order to do this, patients were divi-

    ded into groups according to a reported transition rating

    scale comparing pre- and postoperative health: much bet-

    ter, a little better, about the same, a little worse or much

    worse. Then, the mean change in score for those who

    reported to be a little better minus the mean change in

    score for those who reported to be about the same pro-

    duced the MID [11, 13].

    For the statistical tests, results of p \ 0.05 were con-sidered significant.


    Patient characteristics

    Fifty-six patients were evaluated in the pre-op, PO15d and

    PO90d of rhinoplasty (rhinoplasty group), whereas a hun-

    dred volunteers were included in control group, as shown

    in Table 1. There was no difference in gender or age dis-

    tribution between both groups.

    Data obtained

    Internal consistency of the questionnaire, measured by the

    Cronbachs alpha coefficient, was high (0.86). Eliminating

    one question at time, Cronbachs alpha scores varied from

    0.79 to 0.87, showing homogeneity among questions.

    Testretest reproducibility was evaluated with all

    patients in the pre-op. Inter-examiner and intra-examiner

    ICC were 0.90 and 0.94, respectively, indicating good


    Controls, pre-op and PO15d ROE scores were normally

    distributed according to KolmogorovSmirnov test

    (p = 0.13, p = 0.12 and p = 0.23, respectively). How-

    ever, PO90d ROE scores were not (p = 0.01).

    Validity of the instrument was shown in Table 2. Le-

    venes test showed heterogeneity of variance between

    patients and controls, so MannWhitney U test was per-

    formed. Brazilian Portuguese ROE questionnaire could

    distinguish patients from controls.

    Responsiveness of the instrument was noticed even in a

    very recent postoperative follow-up such as 15 days, as

    presented in Table 3. Moreover, the questionnaire was

    sufficiently sensitive for detecting changes in patients

    satisfaction from PO15d to PO90d. As PO90d ROE scores

    were not normally distributed, Wilcoxon signed-rank test

    was used.

    Table 1 Characteristics of the sample

    Characteristics Rhinoplasty Controls

    Women Men Total Women Man Total

    Number 33 23 56 56 44 100

    Percentage 58.93 % 41.07 % 100 % 56 % 44 % 100 %

    Mean age (years) 27.57 31.16 29.65 30.02 31.39 30.79

    SD of age (years) 9.47 10.00 9.86 7.03 10.42 9.07

    Range of age (years) 1453 1550 1453 2166 1865 1866

    t test: female patients versus male patients, p = 0.18; female controls versus male controls, p = 0.44; total women versus total men, p = 0.48

    SD standard deviation

    Qual Life Res (2014) 23:953958 955


  • The evolution of Brazilian Portuguese ROE scores

    according to the time of the surgery can be seen in Table 4.

    Preoperative patients scores were much lower than

    controls scores. At PO15d, patients have already presented

    improvement in their quality of life, which placed them on

    the same level as control group. At PO90d, rhinoplasty was

    responsible for a greater improvement in patients quality

    of life that led them even better than controls.

    Rhinoplasty effect size on quality of life (measured by

    the mean change score divided by baseline standard devi-

    ation) detected by Brazilian Portuguese ROE was consid-

    ered large (PO15d in relation to pre-op = 3.22; PO90d in

    relation to pre-op = 4.06). Effect size between PO15d and

    PO90d was moderate (0.71).

    The mean score for each question in each group could

    be seen in Table 5. Most of questions have presented sta-

    tistically significant difference among groups, but between

    PO15d and controls, as presented in Table 5.

    To calculate the MID, clinical condition classification

    from both PO15d and PO90d was obtained. Mean changes

    in scores from postoperative to preoperative for each of the

    symptom transition rating groups were showed in Table 6.

    The minimally important difference was of 8.67 points.

    That means that a variation of less than 9 points in the

    Brazilian Portuguese ROE might not be perceived as

    worsening or improvement by the patient.


    When elaborating the ROE, Alsarraf was concerned about

    the ease and simplicity of instrument administration [2].

    After translation and cross-cultural adaptation [7], despite

    difficulties in text reading and comprehension by part of

    the population seen in our service, this characteristic was

    not lost. In this study, questions were read to the patients

    Table 2 Validation of ROE

    ROE score Groups

    Rhinoplasty (pre-op) Controls

    Participants 56 100

    Mean score 7.14 17.94

    SD score 3.29 3.91

    Median score 7 18

    Levenes test: patients versus controls p = 0.04*; MannWhitney

    U test: patients versus controls p \ 0.0001*; effect size (95 %CI) = -2.92 (-3.37 to -2.45)

    ROE Rhinoplasty Outcome Evaluation; pre-op preoperative; SD

    standard deviation; CI confidence interval

    Table 3 Responsiveness of ROE

    ROE score Rhinoplasty

    Pre-op PO15d PO90d

    Average 7.14 17.73 20.50

    SD 3.29 3.88 3.51

    Median 7 18 21

    Levenes test: p = 0.27; KS test: pre-op p = 0.12, PO15d p = 0.23,

    PO90d p = 0.01*; t test: pre-op versus PO15d p \ 0.00001*; Wil-coxon test pre-op versus PO90d p \ 0.000001*; Wilcoxon testPO15d versus PO90d p \ 0.0001*; ES (95 % CI): pre-op versusPO15d 2.94 (2.393.46); pre-op versus PO90d 3.93 (3.274.53);

    PO15d versus PO90d 0.75 (0.361.13)

    ROE Rhinoplasty Outcome Evaluation; pre-op preoperative; PO15d

    15 days postoperative; PO90d 90 days postoperative; ES effect size;

    CI confidence interval

    Table 4 Variation of the patients scores in comparison with the controls

    ROE score Pre-op Controls Variation ROE Levenes test MannWhitney U test Effect size (95 % CI)

    Mean 7.14 17.94 -10.80 p = 0.04* p \ 0.0001* -2.92 (-3.36 to -2.44)SD 3.29 3.91

    Median 7 18

    ROE score PO15d Controls Variation ROE Levenes test t test Effect size (95 % CI)

    Mean 17.73 17.94 -0.21 p = 0.96 p = 0.75 -0.05 (-0.38 to 0.27)

    SD 3.88 3.91

    Median 18 18

    ROE score PO90d Controls Variation ROE Levenes test MannWhitney U test Effect size (95 % CI)

    Mean 20.50 17.94 2.56 p = 0.25 p \ 0.0001* 0.68 (0.341.01)SD 3.51 3.91

    Median 21 18

    KolmogorovSmirnov test: controls p = 0.13, pre-op p = 0.12, PO15d p = 0.23, PO90d p = 0.01*

    ROE Rhinoplasty Outcome Evaluation; pre-op preoperative; PO15d 15 days postoperative; PO90d 90 days postoperative; CI confidence interval

    956 Qual Life Res (2014) 23:953958


  • instead of self-administration, like others Brazilian ques-

    tionnaires studies [7,...


View more >