View
216
Download
1
Category
Preview:
Citation preview
Validation of the Rhinoplasty Outcomes Evaluation (ROE)questionnaire adapted to Brazilian Portuguese
Suemy Cioffi Izu • Eduardo Macoto Kosugi • Alessandra Stanquini Lopes •
Karen Vitols Brandao • Leonardo Bomediano Garcia Sousa • Vinıcius Magalhaes Suguri •
Luis Carlos Gregorio
Accepted: 18 September 2013 / Published online: 1 October 2013
� Springer Science+Business Media Dordrecht 2013
Abstract
Purpose The aim of this study was to validate the Rhi-
noplasty Outcomes Evaluation (ROE) questionnaire adap-
ted to the Brazilian Portuguese.
Method A prospective study was conducted with ROE
administration to 56 patients submitted to rhinoplasty
(preoperatively, and then 15-day and 90-day postopera-
tively) and 100 volunteers without the need or desire of
cosmetic or functional nasal surgery. Reliability (internal
consistency and test–retest reproducibility), validity,
responsiveness and clinical interpretability were assessed.
Results Rhinoplasty patients’ mean preoperative score
was 7.14, 15 days post-op 17.73 and 90 days post-op
20.50, while controls presented 17.94 points (p \ 0.0001),
showing the questionnaire’s validity and responsiveness.
Internal consistency was 0.86. Inter- and intra-examiner
test–retest reproducibility was 0.90 and 0.94, respectively.
The effect size caused by the surgery was considered large
(15 days post-op compared to the preoperative score: effect
size = 3.22; 90 days post-op compared to preoperative
score: effect size = 4.06). The minimally important dif-
ference was 8.67 points, so changes smaller than 9 points in
ROE might not be perceived by the patient as an
improvement or worsening.
Conclusion The Brazilian Portuguese version of ROE is a
valid instrument to assess results in rhinoplasty patients.
Keywords Rhinoplasty � Quality of life �Questionnaires
Introduction
Physicians’ concept of a good surgical outcome could be
very different from patients’ thoughts. Most likely, the
professional experience of surgeons leads to different
expectations compared to patients [1]. A given surgeon
may be satisfied with his or her results, but if patients
themselves are not similarly pleased, then the intervention
cannot totally be considered a success [2]. However,
patient satisfaction could be influenced by many variables,
such as availability and convenience of health care, the
‘‘bedside manner’’ of the doctor, affability of the extended
team and perceived cleanliness of the hospital, which can
complicate evaluation of clinical outcome [3, 4].
In this context, the quality of life questionnaires are very
suitable tools that allow quantitative assessment of other-
wise subjective results, such as patient satisfaction and,
consequently, surgery success [3]. Quality of life can be
defined as ‘‘individuals’ perception of their position in life
in the context of the culture and value systems in which
they live and in relation to their goals, expectations, stan-
dards and concerns. It is a broad ranging concept affected
in a complex way by the persons’ physical health, psy-
chological state, level of independence, social relationships
and their relationship to salient features of their environ-
ment’’ [5]. In aesthetic interventions, more than any other
aspect of rhinology, patient satisfaction and quality of life
must be the measure against which successful procedure
should be judged [3].
Based on such philosophy, Alsarraf developed four new
outcome instruments to assess the results of cosmetic facial
procedures: Rhinoplasty Outcomes Evaluation (ROE),
Facelift Outcomes Evaluation (FOE), Blepharoplasty
Outcomes Evaluation and Skin Rejuvenation Outcomes
Evaluation (SROE) [2, 6]. In order to measure outcomes
S. C. Izu � E. M. Kosugi (&) � A. S. Lopes �K. V. Brandao � L. B. G. Sousa � V. M. Suguri � L. C. Gregorio
Department of Otorhinolaryngology and Head and Neck
Surgery, UNIFESP-EPM, Sao Paulo, SP, Brazil
e-mail: edumacoto@uol.com.br
123
Qual Life Res (2014) 23:953–958
DOI 10.1007/s11136-013-0539-x
such as patient satisfaction and quality of life in the facial
plastic surgery patient, he identified three key aspects that
constituted such satisfaction for each treatment modality of
interest: physical, emotional and social factor. Emotional
and social factors were similar among each of the specific
interventions assessed by Alsarraf [2], while physical fac-
tors presented very specific characteristics depending on
the surgical option.
ROE was translated and cross-culturally adapted to
Brazilian Portuguese [7], according to Guillemin et al. [8].
However, validation process was not carried out. The
objective of this study is to validate the Brazilian Portu-
guese version of the ROE.
Method
This research was approved by the Ethics in Research
Committee of the institution, under protocol number CEP
1791/11, and all participants were volunteers and signed
the informed consent form.
The Brazilian Portuguese ROE questionnaire [7] is
shown in Fig. 1. It is comprised of six questions; each of
the six items was scored on a 0–4 scale, with 0 repre-
senting the most negative response and 4 representing the
most positive response. Therefore, the total score can vary
from 0 to 24. In order to facilitate the comprehension of
the results, the total score can be divided by 24 and
multiplied by 100, so that the score can vary from 0 to
100 %. So, 24 points or 100 % means the most patient
satisfaction [2].
Recruitment of patients
Volunteer patients that desired cosmetic nasal surgery
(rhinoplasty) with or without functional surgery (septo-
plasty and/or turbinectomy) were recruited from a rhinol-
ogy unit to be part of rhinoplasty group. Exclusion criteria
comprised the desire not to participate in the study or
revision cases. Brazilian Portuguese ROE questionnaire
was filled in by participants in the preoperative (pre-op)
and in 15 and 90 days postoperative periods (PO15d and
PO90d, respectively). In the preoperative period, the
questionnaire was administered by two of the authors
(A.S.L. and K.V.B.) and was repeated by one of them after
15 days to check reproducibility. Afterward, the patients
were assessed by one of the authors (A.S.L. or K.V.B.) in
the PO15d and in the PO90d. In the postoperative evalu-
ations, the patients also answered whether they felt: much
better, a little better, about the same, a little worse or much
worse, than the period before the intervention.
The control group was made up of healthy volunteers
with no desire or need for aesthetic and/or functional nasal
surgery, who were recruited from the clinic, employees and
students or relatives from our institution.
Analysis of data
Reliability was analyzed in two ways: internal consistency
and test–retest reproducibility. Internal consistency refer-
red to the way individual items relate to each other, in order
to provide homogeneity among them, and was measured
using Cronbach’s alpha [9]. The minimum acceptable score
Fig. 1 Brazilian Portuguese
version of Rhinoplasty
Outcomes Evaluation
questionnaire
954 Qual Life Res (2014) 23:953–958
123
for Cronbach’s alpha is 0.7 [10–12]. Test–retest repro-
ducibility measured the stability of an instrument over time
after repeated testing [10, 11]. The questionnaire was
applied twice in 15 days by the same examiner (intra-
examiner test–retest reproducibility) and twice in the same
day by two different examiners (inter-examiner test–retest
reproducibility) [10]. The correlation between the scores,
measured by intra-class correlation coefficient (ICC), must
be of, at least, 0.7 [10, 11].
Measure’s validity is the capacity of the questionnaire to
reflect differences among known groups (diseased vs. non-
diseased) using the unpaired t test or Mann–Whitney
U test, depending on samples’ homogeneity of variance
measured by Levene’s test and normal distribution mea-
sured by Kolmogorov–Smirnov test. Basically, Brazilian
Portuguese ROE should be able to distinguish the group of
patients that wish/need rhinoplasty from the group of
subjects that did not [10, 11].
Responsiveness is the ability of the questionnaire to
detect clinical differences over time. Pre- and postoperative
scores were compared using paired t test or Wilcoxon
signed-rank test, depending on samples’ homogeneity of
variance measured by Levene’s test and normal distribu-
tion measured by Kolmogorov–Smirnov test. Also,
responsiveness was assessed by measuring the effect size:
the mean change score divided by baseline standard devi-
ation. By convention, an effect size of [0.2 is considered
small, [0.5 moderate and [0.8 a large improvement in
health quality of life [10, 11].
Clinical interpretability can be calculated by the mini-
mally important difference (MID), the smallest change in
scores that a group of patients can detect as a real
improvement [13]. In order to do this, patients were divi-
ded into groups according to a reported transition rating
scale comparing pre- and postoperative health: much bet-
ter, a little better, about the same, a little worse or much
worse. Then, the mean change in score for those who
reported to be ‘‘a little better’’ minus the mean change in
score for those who reported to be ‘‘about the same’’ pro-
duced the MID [11, 13].
For the statistical tests, results of p \ 0.05 were con-
sidered significant.
Results
Patient characteristics
Fifty-six patients were evaluated in the pre-op, PO15d and
PO90d of rhinoplasty (rhinoplasty group), whereas a hun-
dred volunteers were included in control group, as shown
in Table 1. There was no difference in gender or age dis-
tribution between both groups.
Data obtained
Internal consistency of the questionnaire, measured by the
Cronbach’s alpha coefficient, was high (0.86). Eliminating
one question at time, Cronbach’s alpha scores varied from
0.79 to 0.87, showing homogeneity among questions.
Test–retest reproducibility was evaluated with all
patients in the pre-op. Inter-examiner and intra-examiner
ICC were 0.90 and 0.94, respectively, indicating good
reliability.
Controls, pre-op and PO15d ROE scores were normally
distributed according to Kolmogorov–Smirnov test
(p = 0.13, p = 0.12 and p = 0.23, respectively). How-
ever, PO90d ROE scores were not (p = 0.01).
Validity of the instrument was shown in Table 2. Le-
vene’s test showed heterogeneity of variance between
patients and controls, so Mann–Whitney U test was per-
formed. Brazilian Portuguese ROE questionnaire could
distinguish patients from controls.
Responsiveness of the instrument was noticed even in a
very recent postoperative follow-up such as 15 days, as
presented in Table 3. Moreover, the questionnaire was
sufficiently sensitive for detecting changes in patients’
satisfaction from PO15d to PO90d. As PO90d ROE scores
were not normally distributed, Wilcoxon signed-rank test
was used.
Table 1 Characteristics of the sample
Characteristics Rhinoplasty Controls
Women Men Total Women Man Total
Number 33 23 56 56 44 100
Percentage 58.93 % 41.07 % 100 % 56 % 44 % 100 %
Mean age (years) 27.57 31.16 29.65 30.02 31.39 30.79
SD of age (years) 9.47 10.00 9.86 7.03 10.42 9.07
Range of age (years) 14–53 15–50 14–53 21–66 18–65 18–66
t test: female patients versus male patients, p = 0.18; female controls versus male controls, p = 0.44; total women versus total men, p = 0.48
SD standard deviation
Qual Life Res (2014) 23:953–958 955
123
The evolution of Brazilian Portuguese ROE scores
according to the time of the surgery can be seen in Table 4.
Preoperative patients’ scores were much lower than
controls’ scores. At PO15d, patients have already presented
improvement in their quality of life, which placed them on
the same level as control group. At PO90d, rhinoplasty was
responsible for a greater improvement in patients’ quality
of life that led them even better than controls.
Rhinoplasty effect size on quality of life (measured by
the mean change score divided by baseline standard devi-
ation) detected by Brazilian Portuguese ROE was consid-
ered large (PO15d in relation to pre-op = 3.22; PO90d in
relation to pre-op = 4.06). Effect size between PO15d and
PO90d was moderate (0.71).
The mean score for each question in each group could
be seen in Table 5. Most of questions have presented sta-
tistically significant difference among groups, but between
PO15d and controls, as presented in Table 5.
To calculate the MID, clinical condition classification
from both PO15d and PO90d was obtained. Mean changes
in scores from postoperative to preoperative for each of the
symptom transition rating groups were showed in Table 6.
The minimally important difference was of 8.67 points.
That means that a variation of less than 9 points in the
Brazilian Portuguese ROE might not be perceived as
worsening or improvement by the patient.
Discussion
When elaborating the ROE, Alsarraf was concerned about
the ease and simplicity of instrument administration [2].
After translation and cross-cultural adaptation [7], despite
difficulties in text reading and comprehension by part of
the population seen in our service, this characteristic was
not lost. In this study, questions were read to the patients
Table 2 Validation of ROE
ROE score Groups
Rhinoplasty (pre-op) Controls
Participants 56 100
Mean score 7.14 17.94
SD score 3.29 3.91
Median score 7 18
Levene’s test: patients versus controls p = 0.04*; Mann–Whitney
U test: patients versus controls p \ 0.0001*; effect size (95 %
CI) = -2.92 (-3.37 to -2.45)
ROE Rhinoplasty Outcome Evaluation; pre-op preoperative; SD
standard deviation; CI confidence interval
Table 3 Responsiveness of ROE
ROE score Rhinoplasty
Pre-op PO15d PO90d
Average 7.14 17.73 20.50
SD 3.29 3.88 3.51
Median 7 18 21
Levene’s test: p = 0.27; KS test: pre-op p = 0.12, PO15d p = 0.23,
PO90d p = 0.01*; t test: pre-op versus PO15d p \ 0.00001*; Wil-
coxon test pre-op versus PO90d p \ 0.000001*; Wilcoxon test
PO15d versus PO90d p \ 0.0001*; ES (95 % CI): pre-op versus
PO15d 2.94 (2.39–3.46); pre-op versus PO90d 3.93 (3.27–4.53);
PO15d versus PO90d 0.75 (0.36–1.13)
ROE Rhinoplasty Outcome Evaluation; pre-op preoperative; PO15d
15 days postoperative; PO90d 90 days postoperative; ES effect size;
CI confidence interval
Table 4 Variation of the patients’ scores in comparison with the controls
ROE score Pre-op Controls Variation ROE Levene’s test Mann–Whitney U test Effect size (95 % CI)
Mean 7.14 17.94 -10.80 p = 0.04* p \ 0.0001* -2.92 (-3.36 to -2.44)
SD 3.29 3.91
Median 7 18
ROE score PO15d Controls Variation ROE Levene’s test t test Effect size (95 % CI)
Mean 17.73 17.94 -0.21 p = 0.96 p = 0.75 -0.05 (-0.38 to 0.27)
SD 3.88 3.91
Median 18 18
ROE score PO90d Controls Variation ROE Levene’s test Mann–Whitney U test Effect size (95 % CI)
Mean 20.50 17.94 2.56 p = 0.25 p \ 0.0001* 0.68 (0.34–1.01)
SD 3.51 3.91
Median 21 18
Kolmogorov–Smirnov test: controls p = 0.13, pre-op p = 0.12, PO15d p = 0.23, PO90d p = 0.01*
ROE Rhinoplasty Outcome Evaluation; pre-op preoperative; PO15d 15 days postoperative; PO90d 90 days postoperative; CI confidence interval
956 Qual Life Res (2014) 23:953–958
123
instead of self-administration, like others Brazilian ques-
tionnaires studies [7, 10, 14, 15]. Probably, the adminis-
tration of the questionnaire to patients did not alter its
purpose, since the reading was done ipsis litteris, without
any explanation of the questions. Moreover, this way of
administration had some advantages over self-administra-
tion, such as faster filling out time, lower rate of missing
data and interviewees’ preference [16]. However, an
interviewer can enhance motivation to respond, creating
worse results in preoperative answers and better results in
postoperative ones [17]. Nevertheless, a study with stroke
patients showed that modes of administration of question-
naires could be used interchangeably [18].
Brazilian Portuguese version of ROE questionnaire
showed high internal consistency like the original one, with
Cronbach’s alpha coefficient of 0.86 and 0.84, respectively
[6]. The test–retest reproducibility was assessed in different
ways. In the original study, patients filled out the ques-
tionnaire (self-administration) two times: first, during ini-
tial consult and then on the day of surgery (0.5–9.5 weeks
after the initial consult; mean = 3.5 weeks), with intra-
class correlation coefficient of 0.83 [6]. Alsarraf et al. did
not assess inter-examiner test–retest reproducibility, due to
self-administration of questionnaires. Despite differences
in the administration of questionnaire, high correlation
coefficients have been achieved by both forms.
Regarding the validity of the questionnaire, Brazilian
Portuguese ROE had an optimum performance. A great
difference in scores was noted when comparing subjects
that needed or desired rhinoplasty (mean = 7.14 or
29.75 %) with subjects that did not (mean = 17.94 or
74.75 %), and these results were in agreement with Izu
et al. [7], who obtained a cutoff of 12, separating ‘‘dis-
eased’’ from ‘‘non-diseased’’ subjects.
According to other studies that used ROE questionnaire
as an outcome measure for rhinoplasty [6, 7, 19–23],
postoperative scores were much better than preoperative
ones. In original ROE validation, Alsarraf et al. [6] showed
a mean gain of 44.5 % in ROE scores after a five-month
follow-up, whereas the present study reached 55.66 % of
mean gain after three-month follow-up. PO3 m Group
scores were statistically significant better than control
group scores, indicating that postoperative patients’ satis-
faction could have exceeded that of control group. This
demonstrated the responsiveness of the test, which pre-
sented high effect size too.
Follow-up time varied in the literature [6, 7, 19–23]. In
the present study, differences in satisfaction were noted
between 15 and 90 days follow-up, probably due to
important edema and other complaints presented by
patients at PO15d, which usually improved at PO90d. After
6 months from surgery, less improvement usually occurs,
so that Arima et al. [23] with follow-up time varying from
6 months to 10 years, did not find any statistically signif-
icant difference in quality of life related to the follow-up
time.
When analyzing preoperative scores for each question,
nasal aesthetic issues specifically (questions 1, 5 and 6)
presented worse scores. Question 2, about nasal obstruc-
tion, also had low scores, but better than nasal aesthetic
questions. However, all questions presented good postop-
erative scores, showing that even if primary purpose was
only aesthetic, great attention should be paid to obstructive
factors during rhinoplasty, since this would promote great
improvement in quality of life too.
Although this study presented improvements in patients’
quality of life after cosmetic rhinoplasty, the purpose of
this study was not to evaluate this procedure specifically,
but to demonstrate the ability of Brazilian version of ROE
questionnaire to accurately distinguish these modifications:
Could ROE discern satisfied versus unsatisfied postopera-
tive patients? Satisfied patients are as pleased as people
who do not deserve rhinoplasty? So, comparing pre- and
postoperative patients with healthy controls could be the
best way to measure the exact impairment in quality of life
caused by disease in preoperative point and the gain after
surgery. However, the choice of control group always
Table 5 Mean scores per question per group
Question ROE mean scores
Pre-op PO15d PO90d Controls
1. 0.76 2.88 3.55 2.79
2. 1.53 2.75 3.45 3.03
3. 1.54 3.11 3.61 2.89
4. 2.75 3.52 3.91 3.84
5. 0.46 2.82 3.41 2.36
6. 0.05 2.55 2.48 2.95
t test: significant for all the comparisons between groups for each
question, but for questions 1, 2, 3 and 6 between PO15d and controls;
questions 4 and 6 between PO90d and controls; and question 6
between PO15d and PO90d
ROE Rhinoplasty Outcome Evaluation; pre-op preoperative; PO15d
15 days postoperative; PO90d 90 days postoperative
Table 6 Post–preoperative scores variation per transition group
Symptom transition rating group N Mean variation SD
Much better 94 13.04 4.42
A little better 15 8.67 2.66
The same 1 0 X
A little worse 2 0.5 4.95
Much worse 0 X X
N number; SD standard deviation
Qual Life Res (2014) 23:953–958 957
123
evokes discussion. We would like to choose volunteers
who really do not want or need nasal surgery, so that we
could compare these satisfied volunteers with satisfied
postoperative patients. Now, with a validated instrument,
many settings should be compared, beyond pure cosmetic
rhinoplasty, as malformation and post-trauma reconstruc-
tive rhinoplasty, for instance.
Conclusion
Brazilian Portuguese version of the ROE questionnaire is a
valid instrument to assess results in rhinoplasty patients,
presenting good internal consistency, reproducibility,
validity and responsiveness.
References
1. Moolenburgh, S. E., Mureau, M. A., & Hofer, S. O. (2008).
Aesthetic outcome after nasal reconstruction: Patient versus panel
perception. Journal of Plastic, Reconstructive & Aesthetic Sur-
gery, 61(12), 1459–1464.
2. Alsarraf, R. (2000). Outcomes research in facial plastic surgery:
A review and new directions. Aesthetic Plastic Surgery, 23(3),
192–197.
3. Hopkins, C. (2009). Patient reported outcome measures in rhi-
nology. Rhinology, 47(1), 10–17.
4. Fitzpatrick, R. (1991). Surveys of patient satisfaction : I-impor-
tant general considerations. British Medical Journal, 302(6781),
887–889.
5. The WHOQOL Group. (1998). The World Health Organization
Quality of Life Assessment (WHOQOL): Development and
general psychometric properties. Social Science and Medicine,
46(12), 1569–1585.
6. Alsarraf, R., Larrabee, W. F., Anderson, S., Murakami, C. S., &
Johnson, C. S, Jr. (2001). Measuring cosmetic facial plastic
surgery outcomes: A pilot study. Archives of Facial Plastic
Surgery, 3(3), 198–201.
7. Izu, S. C., Kosugi, E. M., Brandao, K. V., Lopes, A. S., Garcia, L.
B., Suguri, V. M., et al. (2012). Normal values for the Rhino-
plasty Outcome Evaluation (ROE) questionnaire. Brazilian
Journal of Otorhinolaryngology, 78(4), 76–79.
8. Guillemin, F., Bombardier, C., & Beaton, D. (1993). Cross-cul-
tural adaptation of health-related quality of life measures: Liter-
ature review and proposed guidelines. Journal of Clinical
Epidemiology, 46(12), 1417–1432.
9. Cronbach, L. J. (1951). Coefficient alpha and the internal struc-
ture of tests. Psychometrika, 16(3), 297–334.
10. Kosugi, E. M., Chen, V. G., Fonseca, V. M., Cursino, M. M.,
Mendes-Neto, J. A., & Gregorio, L. C. (2011). Translation, cross-
cultural adaptation and validation of SinoNasal Outcome Test
(SNOT): 22 to Brazilian Portuguese. Brazilian Journal of Oto-
rhinolaryngology, 77(5), 663–669.
11. Hopkins, C., Gillett, S., Slack, R., Lund, V. J., & Browne, J. P.
(2009). Psychometric validity of the 22-item Sinonasal Outcome
Test. Clinical Otolaryngology, 34(5), 447–454.
12. Scientific Advisory Committee of the Medical Outcomes Trust.
(2002). Assessing health status and quality-of-life instruments :
Attributes and review criteria. Quality of Life Research, 11(3),
193–205.
13. Jaeschke, R., Singer, J., & Guyatt, G. H. (1989). Measurement of
health status. Ascertaining the minimal clinically important dif-
ference. Controlled Clinical Trials, 10(4), 407–415.
14. Klatchoian, D. A., Len, C. A., Terreri, M. T., Silva, M., Itamoto,
C. H., Ciconelli, R. M., et al. (2008). Quality of life of children
and adolescents from Sao Paulo: reliability and validity of the
Brazilian version of the Pediatric Quality of Life Inventory ver-
sion 4.0 Generic Core Scales. Jornal de Pediatria, 84(4),
308–315.
15. Fonseca, E. S., Camargo, A. L., Castro, R. A., Sartori, M. G.,
Fonseca, M. C., Lima, G. R., et al. (2005). Validation of a quality
of life questionnaire (King0s Health Questionnaire) in Brazilian
women with urinary incontinence. Revista Brasileira de Ginec-
ologia e Obstetrıcia, 27(5), 235–242.
16. Weinberger, M., Oddone, E. Z., Samsa, G. P., & Landsman, P. B.
(1996). Are health-related quality-of-life measures affected by
the mode of administration? Journal of Clinical Epidemiology,
49(2), 135–140.
17. Bowling, A. (2005). Mode of questionnaire administration can
have serious effects on data quality. Journal of Public Health
(Oxford, England), 27(3), 281–291.
18. Sutton, C. J., Watkins, C. L., Cook, N., Leathley, M. J., McAdam,
J., & Dey, P. (2013). Postal and face-to-face administration of
stroke outcome measures: Can mixed modes be used? Stroke,
44(1), 217–219.
19. Hellings, P. W., & Nolst Trenite, G. J. (2007). Long-term patient
satisfaction after revision rhinoplasty. The Laryngoscope, 117(6),
985–989.
20. Arima, L. M., Velasco, L. C., & Tiago, R. S. (2012). Influence of
age on rhinoplasty outcomes evaluation: A preliminary study.
Aesthetic Plastic Surgery, 36(2), 248–253.
21. Faidiga, G. B., Carenzi, L. R., Yassuda, C. C., Silveira, F., Lago,
T. D., Leite, M. G., et al. (2010). Long-term evaluation in aes-
thetic rhinoplasty in an academic referral center. Brazilian
Journal of Otorhinolaryngology, 76(4), 437–441.
22. Meningaud, J. P., Lantieri, L., & Bertrand, J. C. (2008). Rhino-
plasty: An outcome research. Plastic and Reconstructive Surgery,
121(1), 251–257.
23. Arima, L. M., Velasco, L. C., & Tiago, R. S. (2011). Crooked
nose: Outcome evaluations in rhinoplasty. Brazilian Journal of
Otorhinolaryngology, 77(4), 510–515.
958 Qual Life Res (2014) 23:953–958
123
Recommended