Nutritional Epidemiology

Relative Validity of Food Intake Estimates Using a Food FrequencyQuestionnaire Is Associated with Sex, Age, and OtherPersonal Characteristics1,2

Geoffrey C. Marks,*3 Maria Celia Hughes,y and Jolieke C. van der Pols*y

* School of Population Health, University of Queensland, Herston, Qld 4006, Australia andyQueensland Institute of Medical Research, Herston, Qld 4029, Australia

ABSTRACT We investigated the validity of food intake estimates obtained by a self-administered FFQ relative toweighed food records (WFR) and the extent to which demographic, anthropometric, and social characteristics explaindifferences between these methods. A community-based sample of 96 Australian adults completed a FFQ and 12 d ofWFR over 12 mo. The FFQ was adapted to the Australian setting from the questionnaire used in the US Nurses’Health Study. Spearman rank correlation coefficients ranged from 0.08 for ‘‘other vegetables’’ to 0.88 for tea. Exactagreement by quartiles of intake ranged from 27% (eggs) to 63% (tea). Differences between FFQ and WFRregressed on personal characteristics were significantly associated with at least 1 characteristic for 20 of the 37foods. Sex was significantly associated with differences for 17 food groups, including 5 specific vegetable groups and2 ‘‘total’’ fruit and vegetable groups. Use of dietary supplements and the presence of a medical condition wereassociated with differences for 5 foods; age, school leaving age, and occupation were associated with differences for1–3 foods. BMI was not associated with differences for any foods. Regression models explained from 3% (whole-meal bread) to 37% (for all cereals and products) of variation in differences between methods. We conclude that therelative validity of intake estimates obtained by FFQ is different for men and women for a large number of foods.These results highlight the need for appropriate adjustment of diet-disease relations for factors affecting the validity offood intake estimates. J. Nutr. 136: 459–465, 2006.

KEY WORDS: � food frequency questionnaire � weighed diet records � validation � relative validity� dietary assessment

There is an increasing need for reliable measurements offoods that are consumed as part of the usual diet. Valid esti-mates of the consumption of food items are the premise ofdietary pattern characterizations, and intervention trials suchas the CARET study showed that whole foods rather thanindividual nutrients may best indicate the potential role of thediet in disease prevention (1).

FFQ are widely used to investigate customary food intakeover extended periods of time. Like all dietary methods,estimates derived from FFQ data suffer from random andsystematic error and may not represent the ‘‘true’’ usual diet.Numerous factors may compromise the validity of food con-sumption estimates (2), but the effects of these on measure-ment error are generally assessed for nutrient intake estimatesrather than food intake per se. As a result, the validity of foodintake estimates derived from FFQ data is not well docu-

mented. Further, experience with validation of nutrient intakeestimates showed that specific subject characteristics arefrequently associated with measurement error (3,4). Knowledgeof such factors may help improve the design of food intakeinstruments and provide a basis for more appropriate modeling ofdiet-disease relations.

Here we evaluate the relative validity of food intakeestimates derived from an FFQ administered to participantsin the Nambour Skin Cancer Prevention Trial (5). This fieldtrial was conducted in an unselected adult population in Australia.One of the central objectives of the project was to examine therelation of dietary factors to development of actinic skin and eyedisease (6). This study compares estimates of intake of food fromthe FFQ with those based on 12 d of weighed food records (WFR)over a 12-mo period for a randomly selected subsample of theNambour study population. We estimated the relative bias andimprecision of food intake estimates, and assessed the extent towhich selected demographic, anthropometric, and social charac-teristics of participants explain any difference between the 2dietary methods.

SUBJECTS AND METHODS

Selection of study subjects. The Nambour Skin Cancer Pre-vention Trial is a community-based randomized trial of the effects of

1 M.C.H. was funded by the Public Health Research and DevelopmentCommittee of the National Health and Medical Research Council of Australia andthe World Cancer Research Fund International. J.C.P. was funded by a NHMRCCapacity Building Grant in Population Health Research (grant number: 252834).

2 Supplemental Tables 1–2 are available as Online Supporting Material withthe online posting of this paper at www.nutrition.org.

3 To whom correspondence should be addressed. Email: g.marks@sph.uq.edu.au.

0022-3166/06 $8.00 � 2006 American Society for Nutrition.

Manuscript received 23 August 2005. Initial review completed 19 September 2005. Revision accepted 22 November 2005.

459

by guest on July 1, 2012jn.nutrition.org

Dow

nloaded from

1.html http://jn.nutrition.org/content/suppl/2006/01/18/136.2.459.DCSupplemental Material can be found at:

daily consumption of a dietary supplement of b-carotene andapplication of sunscreen in the prevention of skin cancer (6). TheNambour Trial began in 1992 with 1621 participants aged 25–75 ywho were previously randomly selected from the Nambour electoralroll for a skin cancer prevalence survey in 1986 (5). At the start of thetrial, trained research assistants explained the objective of the FFQand instructed the participants on how to complete it.

Also in 1992, a random sample of 168 participants was invited toparticipate in a validation study involving WFR. Participants were eligibleto participate if they had completed the FFQ and remained an activeparticipant in the trial (7). All participants provided written informedconsent and the institutional ethics committee approved the study.

Administration of the food frequency questionnaire. The self-administered semiquantitative FFQ was adapted from the question-naire used in the US Nurses’ Health Study developed by Willett et al.(8,9). Revisions were made to ensure that the list of foods reflected theAustralian diet according to the 1983 National Dietary Survey ofAdults (10,11). For the Nambour Trial, further revisions were made toimprove estimates of intake of antioxidant-rich foods (inclusion ofmajor food sources, particularly vegetables and fruits).

The FFQ collected consumption information for 129 food items orfood groups. Respondents were requested to recall how often, onaverage, they consumed a given amount of each food during the past 6mo (judged appropriate for this population). The amounts were inhousehold or common measures such as 1 slice, 1 tablespoon (15 mL),or 1 cup (250 mL), representing 1 standard serve for each food. Responseoptions ranged from ‘‘Never’’ to ‘‘41 times a day.’’ For seasonal fruits andvegetables, participants were asked to indicate how often these foodswere eaten in season. Additional information collected includedcooking methods and specific types of oil, margarine, butter, cereals,and take-out foods eaten. The FFQ also collected information on brand,dosage, and frequency of use of dietary supplements. All FFQ data weredouble entered, and any discrepancies resolved by reference to theoriginal forms.

Administration of the weighed food records. WFR participantscompleted 2 nonconsecutive days of food weighing every 2 mo overa period of 12 mo. Initial start days for data collection were randomlyallocated among participants to ensure that each day of the week wasequally represented and that the records for the sample were spacedevenly over the initial 2-mo block. For subsequent blocks, recordingdays were advanced by 1 d. If the days specified were unsuitable forparticipants, alternative days were determined to ensure an overallbalance of week and weekend days.

Participants used 2-kg capacity digital scales in 2-g gradations toweigh all food and beverages consumed for the 2 recording days. In-formation on recipes and dietary supplements used was also recorded.A research dietitian collected the food diaries and reviewed the re-cords with the participants to check for errors, omissions, or doubtfulentries. Coding decisions were made by the research dietitians whochecked all decisions for open-ended questions in the FFQ andchecked a random 10% subsample of daily records for the WFR (errorrate 0.7%).

Collection of health indicators. At baseline, trained research staffmeasured body weight and height using standard protocols. Informationon age, sex, education, occupation, and medical condition was obtainedby questionnaires (6). Participants were considered to have a medicalcondition if they answered ‘‘yes’’ to any of the conditions listed in thequestion ‘‘Have you ever been told by a doctor/nurse that you have:glaucoma, gallstones, high cholesterol, high triglycerides, diabetes/highblood sugar, high blood pressure/hypertension, angina, heart attack,stroke, cancer?’’

Calculation of food intakes. Frequency of consumption of eachfood item in the FFQ was converted to intake in grams per day bymultiplying the standard serving size of each food as specified in the FFQby the following values for each frequency option: Never5 0;,1/mo50.02; 1–3/mo 5 0.07; 1/wk 5 0.14; 2–4/wk 5 0.43; 5–6/ wk 5 0.79;1/d 5 1.0; 2–3/d 5 2.5; and 41/d 5 4. Seasonal foods were weightedaccording to the proportion of the year that each food was available. Theintended use of the FFQ includes the association of food groups withskin cancer. For this reason, the 129 FFQ food items were reclassifiedinto 37 food groups, including those that reflect dietary patterns thatare hypothesized to modify the risk of skin cancers (see Supplemental

Table 1). Daily grams of intake for individual FFQ items were summedto obtain daily intake of each food group. Data from WFR were enteredinto a database using Xyris Diet 1 Software (7,12). Individual food itemswere classified into 45 food groups. Only the 37 food groups thatmatched the FFQ were included in this analysis. The 8 nonmatchingfood groups included water (drinking or used in food preparation),drinking chocolate, organ meats other than liver, meat pastes, sauces,condiments, meal replacements (e.g., Sustagen�), and meat replace-ment foods. Daily grams of consumption of each food group werecalculated by summing foods in each food group per day of WFR andobtaining the mean of all weighing days.

Statistical analyses. Mean, SD and median food group intakes(grams) were calculated for each dietary method. The intake estimateswere not normally distributed.

Statistical analyses were performed in 2 phases. In the first phase,we compared agreement between the 2 methods by using the intakeestimates in the form in which they would be used in future diet-disease analyses, i.e., (untransformed) grams of intake and quartiles ofgrams of intakes. For this comparison we used nonparametric methodsincluding Spearman rank correlations and the Wilcoxon signed ranktest for difference between paired observations. We compared theclassification of intakes into quartiles by the 2 methods. The pro-portion of exact agreement, deviation by 1 quartile, and the proportionof grossly misclassified individuals (disagreement by 3 quartiles) werecalculated. Finally, we calculated the median grams of food intake asdetermined using WFR for each FFQ quartile. These analyses com-pared the ranking of individuals by food intake.

In the second phase, agreement in estimating absolute estimates ofintakes was assessed using parametric tests for which food group intakedata were log-(natural) transformed (FFQlog and WFRlog) to achievea normal distribution. The formula log(gram intake 1 1) was used iffood groups were not consumed by all participants. Paired t test (P 50.05) and limits of agreement (LOA) described by Bland and Altman(13) with correction for a small sample size (n , 100) by Ludbrook(14) were used to quantify the degree of agreement/disagreement be-tween the 2 dietary methods. Results of paired t tests indicate whetheron average, the FFQ consistently over- or underestimated the WFR.The LOA is calculated as follows:

mean difference6 tn21;0:05SDrð111=nÞ:

The lower and upper boundaries of the LOA present the range inwhich 95% of the differences between the dietary methods wereexpected to lie. Mean differences and LOA were exponentiated toprovide a ratio of the gram intakes estimated by FFQ relative to theWFR (13). Thus, a mean ratio of 1.10 and LOA of 0.85–1.40 indicatethat, on average, FFQ overestimates WFR by 10% and that 95% of thedifferences range from 15% below to 40% above.

The difference in intakes (FFQlog 2 WFRlog) was plotted againstthe mean (FFQlog-adj 1 WFRlog-adj)/2 to determine whether the dif-ference between the methods varied across the range of intakes. Aregression line was fitted and the slope was tested for significant dif-ference from 0 (P 5 0.05). Slopes significantly different from 0 iden-tified cases in which the difference between methods increased ordecreased across the range of intakes.

To identify factors associated with the validity of FFQ intakeestimates, multivariable regression analysis was performed with the dif-ference in log-transformed food intakes between dietary methods asthe dependent variable and personal characteristics of participants asexplanatory variables. Because of the dissimilarity in construct anderror source of the 2 dietary methods, one would generally expect dif-ferences to be random relative to other factors. If any factors are as-sociated with the differences, it would indicate that these factors areassociated with the relative validity of the measures. Personal char-acteristics assessed included age (y), sex, BMI, education (school-leaving age), occupation (professionals, nonprofessionals), medicalcondition (yes, no), and use of dietary supplements (yes, no). Meanintakes were included in the model as a predictor of difference inintake estimates if the preliminary analyses (see above) showed thatmean intakes were associated with the difference in intakes at P ,0.10. R2 was calculated to quantify the extent to which the ex-planatory variables accounted for total variation in the difference inintakes. All statistical tests were two-sided and a significance level of

460 MARKS ET AL.

by guest on July 1, 2012jn.nutrition.org

Dow

nloaded from

P , 0.05 was used. All analyses were performed using SAS v8.2software(15).

RESULTS

A total of 1447 participants (89%) completed the FFQ,whereas a final sample of 115 (69% acceptance rate) completedthe WFR. Of the 115 participants in the WFR, 19 were omittedfrom analyses; 12 completed ,10 d of the WFR, 5 omittedconsumption frequencies for .10% of items in the FFQ, 1woman became pregnant early in the study, and 1 man wasexcluded because his energy intake exceeded limits suggestedby Willett (9). Thus, the sample size for the validity study was96, consisting of 37 men and 59 women.

The 96 subjects in the validation study, the 115 subjects whocompleted the WFR, and the 1621 Nambour Trial populationdid not differ in age, sex, BMI, education, or smoking status.There were significantly more regular users of dietary supple-ments in the validation study (45.8%) compared with the trialparticipants (32.7%) (P, 0.05). The proportion of professionalsor participants with a medical condition did not differ.

The mean and median grams of food group intakes arepresented in Table 1. For 21 of the 37 food groups, estimatedintakes by the FFQ were significantly higher than those byWFR. WFR consumption estimates were significantly higherfor 5 groups, whereas 11 groups did not differ. It is notable thatmean FFQ intakes of all fruit and vegetable food groups (exceptlegumes) were significantly higher, and the SD for most wasalso much larger.

Table 2 presents the extent of agreement in assignment byquartiles of intake for each food group, compared with expectedproportions using random allocation. Exact agreement rangedfrom 27% (eggs) to 63% (tea). Random allocation would resultin 25% exact agreement. There was no gross misclassification(differing by 3 quartiles) for unmodified dairy, alcoholic bever-ages, tea, and coffee. Of the participants, .10% were grosslymisclassified (differing by 3 quartiles) by the FFQ for all veg-etables (no potato), rice, and nuts.

Spearman rank correlation coefficients ranged from 0.08 for‘‘other vegetables’’ to 0.88 for tea. Food groups with correlations$0.70 included all dairy products (unmodified and modified),alcoholic beverages, tea, and coffee. There were 10 food groups

TABLE 1

Intakes from food groups in a semiquantitative FFQ and 12-d WFR kept by a sample of

Nambour residents aged 25 to 75 y, n 5 96.

FFQ WFR

Food group Mean 6 SD Median Mean 6 SD Median

g/dFats and oils 17.3 6 9.7* 15.4 21.2 6 10.6 21.0Unmodified dairy 209.6 6 222.5 126.7 181.9 6 158.3 141.5Modified dairy 181.3 6 235.7* 49.4 113.5 6 131.8 48.5Meat 77.6 6 45.3* 75.9 58.1 6 44.0 48.5Poultry 21.2 6 18.9* 14.0 29.3 6 25.9 23.5Processed meats 28.1 6 44.5 19.3 27.1 6 24.4 24.4Eggs 15.5 6 14.1 10.9 19.7 6 17.3 15.9Seafood 22.7 6 22.2 15.6 19.2 6 21.0 13.5Green leafy vegetables 18.8 6 17.6* 15.3 13.7 6 16.1 10.0Cruciferous vegetables 57.4 6 46.9* 46.5 25.4 6 20.4 20.7Red and yellow vegetables 187.1 6 105.7* 171.4 75.9 6 39.8 68.5Legumes 12.8 6 21.6 6.2 17.2 6 37.0 4.0Potato 113.7 6 54.0* 114.1 72.3 6 40.4 62.0Other vegetables 82.2 6 48.8* 76.2 69.0 6 46.9 62.0All vegetables 472.1 6 186.2* 429.9 273.5 6 133.5 246.0All vegetables (no potato) 358.4 6 163.4* 315.7 201.2 6 112.7 170.8Vitamin A- or C-rich fruits 167.2 6 168.0* 117.5 75.5 6 69.8 61.2Other fruits 293.5 6 216.4* 243.6 137.1 6 84.0 126.0All fruits 460.7 6 336.7* 394.8 212.6 6 130.4 202.5All vegetables and fruit 932.8 6 445.1* 829.3 486.1 6 232.3 455.6All vegetables and fruit (no potato) 819.1 6 430.9* 724.2 413.8 6 217.2 381.0Bread, whole-meal 50.4 6 43.0 30.0 55.4 6 43.1 48.9Bread, white 14.2 6 27.0* 0.6 37.8 6 41.6 20.3Rice 20.3 6 18.3* 14.4 13.4 6 20.4 6.3Pasta and noodles 15.7 6 15.7 10.5 14.9 6 21.1 6.7Breakfast cereals 64.4 6 79.6 34.6 52.5 6 59.6 34.2Other cereals 3.3 6 4.9* 2.6 9.6 6 9.6 6.7All cereals and products 168.2 6 94.0* 140.1 183.6 6 83.6 158.2Cakes and biscuits 39.7 6 32.6* 31.8 50.7 6 41.3 42.1Snacks 3.9 6 8.4* 1.0 2.4 6 5.8 0.0Juice 72.3 6 81.5 53.8 67.9 6 91.8 36.7Alcohol 186.4 6 365.4* 23.7 162.8 6 329.6 12.3Tea 487.8 6 368.9* 625.0 404.4 6 360.8 356.0Coffee 340.9 6 378.4* 250.0 203.5 6 227.5 129.1Soft drinks 144.4 6 230.7* 52.9 69.3 6 133.7 28.8Sugars, sweets, jams 28.9 6 21.7 24.9 30.4 6 25.9 25.1Nuts 8.0 6 10.7* 3.2 4.4 6 6.1 2.1

* Different from WFR, P , 0.05 level (Wilcoxon signed rank test for difference between pairedobservations).

461FACTORS ASSOCIATED WITH VALIDITY OF FFQ FOOD ESTIMATES

by guest on July 1, 2012jn.nutrition.org

Dow

nloaded from

with correlations #0.30; 5 of these were vegetable food groups.The strength of correlations was reflected in the comparison ofmedian WFR intakes per FFQ quartile. Overall, median WFRintakes increased over increasing FFQ quartiles, with a morepronounced increase in intakes observed for correlations$0.50.

The mean ratio (and 95% CI) of FFQ to WFR andLOA was similar to that presented in Table 1, with higherestimates for the FFQ than the WFR in a majority of the foodgroups (see Supplemental Table 2). FFQ estimations of 3 fruitfood groups, 2 vegetable food groups, rice, and pasta andnoodles were double the WFR estimates (mean ratio .2.00),whereas consumption of soft drinks was overestimated by300%. Intakes of fats and oils, whole-meal and white bread,other cereals and cakes, and biscuits were underestimated bythe FFQ. The most marked underestimation was of whitebread; on average, the FFQ estimates were 75% below theWFR.

Although the population mean ratio of FFQ to WFRprovides an estimate of over- or underestimation by the FFQ forthe entire study population, the LOA provides information onthe variability of estimates between the 2 methods at theindividual level. For example, the mean ratio for meat showedoverestimation by 41% (95% CI 17–71%), but the LOAindicated that for 95% of individuals, the differences rangedfrom underestimation at 0.22 of the WFR value to a 9-foldoverestimation.

For 14 food groups, the differences in intakes varied sig-nificantly with the magnitude of the intakes. In 9 food groups,there was a negative association in which the differences werelarger at lower levels of intake, i.e., individuals who do notconsume these food groups regularly showed more error in theirreporting of small amounts of intake compared who those whoate large amounts of the same food group. Differences in es-timated intakes of fats and oils, modified and unmodified dairy,and all cereals and products increased significantly with the

TABLE 2

Comparison of daily grams of food intake obtained from a semiquantitative FFQ and the mean of 12-d WFR kept by a sample of

Nambour residents aged 25 to 75 y, n 5 96.

Allocation by quartilesSpearman correlation

coefficient2

Median intake from WFR by quartile of FFQ intake

Food group Exact Adjacent GM1 Q1 Q2 Q3 Q4

% gFats and oils 39 39 7 0.32 15 17 24 22Unmodified dairy 51 42 0 0.75 52 79 181 294Modified dairy 58 35 3 0.73 0 18 113 229Meat 40 42 5 0.49 23 38 58 67Poultry 36 36 9 0.30 13 21 24 35Processed meats 42 48 2 0.60 4 22 30 37Eggs 27 47 9 0.26 10 20 15 21Seafood 47 31 4 0.46 0 8 18 20Green leafy vegetables 40 40 3 0.40 5 13 13 15Cruciferous vegetables 29 38 6 0.21 18 19 20 29Red and yellow vegetables 31 45 6 0.30 61 65 83 74Legumes 41 33 8 0.39 1 3 6 13Potato 35 36 8 0.35 52 53 69 77Other vegetables 28 29 7 0.08NS 53 73 43 63All vegetables 30 39 9 0.17 225 248 253 251All vegetables (no potato) 30 41 11 0.14NS 160 177 205 160Vitamin A- or C-rich fruits 39 44 4 0.44NS 18 55 78 103Other fruits 45 32 5 0.47 68 116 151 156All fruits 44 35 6 0.48 113 181 222 277All vegetables and fruit 35 43 3 0.43 370 444 453 564All vegetables and fruit(no potato)

36 33 4 0.42 299 389 366 460

Bread, whole-meal 48 32 3 0.56 12 53 57 107Bread, white 49 40 2 0.65 4 14 53 64Rice 30 30 14 0.13NS 5 7 12 7Pasta and noodles 35 31 3 0.28 0 13 9 31Breakfast cereals 42 48 1 0.68 8 38 45 72Other cereals3 41 49 NA4 0.31 2 7 12All cereals and products 31 41 3 0.34 146 147 202 185Cakes, biscuits 41 47 1 0.62 21 37 46 87Snacks3 50 34 NA 0.51 0 0 3Juice 41 49 1 0.65 0 27 40 110Alcohol 54 38 0 0.85 0 2 29 385Tea 63 33 0 0.88 0 345 544 710Coffee 54 38 0 0.81 0 89 290 413Soft drinks 42 45 2 0.61 0 5 82 104Sugars, sweets, jams 48 39 3 0.62 7 20 35 49Nuts 38 33 13 0.22 1 2 2 5Random expected 25 38 13

1 Gross misclassification, disagreement by 3 quartiles.2 All correlation coefficients are significant (P , 0.05) except those identified with NS (¼ not significant).3 Based on tertiles only as frequency distribution of intakes of these items precluded the formation of quartiles.4 NA, not applicable.

462 MARKS ET AL.

by guest on July 1, 2012jn.nutrition.org

Dow

nloaded from

magnitude of intakes. For these food groups, the greatest erroroccurred in subjects who reported high levels of intake com-pared with nonregular consumers of the same food.

Table 3 shows the associations between personal character-istics of individuals and the difference in estimated intakebetween the 2 methods. The difference in reported intakes waslarger among women than men for fats and oils, modified dairy,meat (including processed), poultry and seafood, a number ofvegetable food groups, including green leafy vegetables, allvegetables and fruits, rice, pasta and noodles, and all cerealsand products, whereas the difference was smaller for the intakeof tea. The presence of any medical condition was significantlyassociated with increased difference in reported intakes ofmeat, cruciferous vegetables, pasta and noodles, and tea anddecreased difference for all cereals and products. For example,

the ratio of intakes of meat estimated by FFQ to WFR was 1.51times greater (exponential of 0.412) in participants with med-ical condition than in those with no medical condition. Smallerdifferences in estimated intakes of other vegetables, all veg-etables, and other fruits were noted for participants who reportedtaking dietary supplements, whereas differences were larger foralcoholic beverages and nuts. Although only the difference inintakes of other vegetables, all vegetables, and all vegetables andfruit decreased significantly with the use of dietary supplements,the association between dietary supplements and intake differ-ences in almost all of the other vegetable and fruit food groupswas in the same direction. Age was significantly associated withgreen leafy vegetables, other vegetables, and all cereals andproducts. However, the direction of the associations was notconsistent. BMI was not significantly associated with difference

TABLE 3

Factors associated with difference in (log-transformed) daily grams of food intakes from a semi-quantitative FFQ and 12-d WFR:

regression models were multivariable, with each factor adjusted for all others, n 5 96.

Association between factors and difference in intake estimates (regression coefficients)

Food group Age1 Sex2 BMI1School leaving

age1 Occupation3Medicalcondition4

Use of dietarysupplements5

Mean of FFQand WFR6 R2

Fats and oils 0.001 0.53511 0.003 0.038 0.147 0.288 0.260 0.33711 0.25Unmodified dairy 0.004 0.142 �0.048 �0.097 �0.053 0.335 0.230 0.30011 0.17Modified dairy 0.009 0.7861 0.003 �0.086 0.9051 0.075 0.165 0.1751 0.15Meat 0.004 0.4421 �0.015 �0.054 �0.175 0.4121 0.114 0.15Poultry 0.009 0.70011 0.038 �0.107 0.155 �0.127 �0.108 �0.525111 0.25Processed meats 0.008 0.4851 �0.009 0.093 �0.135 0.062 �0.370 0.09Eggs �0.003 �0.230 0.033 0.049 �0.151 0.065 0.361 �0.48811 0.12Seafood 0.006 0.5291 0.004 �0.118 0.087 0.400 0.067 �0.468111 0.26Green leafy vegetables 0.0211 0.4521 0.062 �0.059 0.143 �0.372 �0.414 0.15Cruciferous vegetables �0.005 0.438 0.040 0.057 �0.015 0.5921 0.221 0.10Red and yellow vegetables �0.005 0.274 0.018 0.010 �0.209 0.177 �0.203 0.12Legumes 0.027 0.450 0.058 �0.139 �0.263 0.304 �0.621 �0.222 0.19Potato 0.010 0.45411 0.002 �0.023 0.045 0.171 �0.175 0.20Other vegetables �0.0191 0.4521 0.040 �0.107 �0.060 0.230 �0.3441 0.23All vegetables �0.001 0.498111 0.029 �0.028 �0.077 0.129 �0.2711 0.28All vegetables (no potato) �0.006 0.44311 0.034 �0.0551 �0.119 0.175 �0.229 �0.243 0.28Vitamin A- or C-rich fruits 0.001 0.152 0.054 �0.045 �0.380 �0.055 0.102 �0.3371 0.14Other fruits 0.002 0.052 �0.012 �0.1651 �0.237 �0.046 �0.3731 0.10All fruits 0.000 0.200 0.014 �0.061 �0.221 �0.072 �0.126 0.08All vegetables and fruit �0.000 0.380111 0.020 �0.046 �0.120 0.069 �0.1931 0.23All vegetables and fruit(no potato)

�0.004 0.32911 0.026 �0.058 �0.161 0.071 �0.188 0.20

Bread, whole-meal 0.011 0.290 0.011 �0.037 �0.051 �0.124 �0.259 0.03Bread, white 0.005 0.620 �0.002 0.163 0.336 �0.268 0.541 0.10Rice 0.013 0.6711 0.071 0.2551 0.146 0.206 0.312 �0.712111 0.31Pasta and noodles �0.028 0.7461 �0.020 0.2991 �0.518 0.92211 �0.089 �0.819111 0.32Breakfast cereals 0.011 0.126 0.009 �0.060 �0.372 0.058 0.242 0.07Other cereals 0.011 0.267 0.031 0.106 0.057 0.228 0.274 �0.3811 0.12All cereals and products 0.0101 0.485111 0.011 �0.019 �0.003 �0.2181 0.112 0.628111 0.37Cakes, biscuits 0.000 0.098 �0.050 0.065 0.263 0.213 �0.042 0.08Snacks �0.006 �0.059 �0.008 �0.037 �0.312 �0.230 �0.077 0.05Juice �0.014 0.312 0.050 �0.009 �0.111 0.205 �0.571 �0.165 0.12Alcohol 0.004 0.584 0.012 �0.042 0.180 0.306 0.5831 0.018 0.13Tea 0.011 �0.75911 �0.052 �0.139 �0.080 0.5871 0.002 0.18Coffee 0.007 �0.295 �0.076 �0.126 �0.005 0.115 �0.042 0.06Soft drinks �0.007 �0.324 0.027 �0.235 0.052 �0.064 0.026 �0.2221 0.08Sugars, sweets, jams �0.004 0.037 �0.041 �0.168 0.097 �0.198 �0.153 0.07Nuts 0.014 �0.092 0.030 0.129 0.346 �0.446x 0.76511 0.17

1P , 0.05; 11P , 0.01; 111P , 0.001.1 Continuous variable.2 Reference category: men.3 Reference category: nonprofessionals.4 Reference category: no medical condition.5 Reference category: no use dietary supplement use.6 Included only for multivariable models in which the mean of the intake estimate was significantly associated with difference in intakes (see Table 3).

463FACTORS ASSOCIATED WITH VALIDITY OF FFQ FOOD ESTIMATES

by guest on July 1, 2012jn.nutrition.org

Dow

nloaded from

in intakes of any food group, but the majority of associations waspositive, i.e., an increasing difference between the 2 dietarymethods with increasing age or BMI.

The R2 statistic gives an indication of the extent to whichthe personal characteristics of participants explained the varia-tion in difference in nutrient intakes between the FFQ andWFR. R2 ranged from as low as 3% up to 37%, explaining 20%or more of the variance for 12 of the 37 food groups. None ofthe personal characteristics were significant in the models forthose with lowest R2, such as for whole-meal bread, snacks,coffee, sugars/sweets/jams, and breakfast cereals. However, sex,medical condition, supplement usage, and age and school leav-ing age together explained ,20% of the variance for fats andoils, poultry, seafood, other vegetables, all vegetables, all veg-etables (no potatoes), all vegetables and fruits, rice, pasta andnoodles, and all cereals and products. Sex was significant for 17of 37 food groups, medical condition, and supplement usage (5food groups each), and age and school leaving age (3 food groupseach). BMI was not associated with intake for any food group.However, after inclusion of energy intake in the model (meanenergy estimated from WFR and FFQ), BMI was positively as-sociated with differences for intake estimates of green leafyvegetables, other vegetables, all vegetables, all vegetables (nopotato) and all vegetables and fruit (not shown).

DISCUSSION

We tested whether a widely used questionnaire that wasdeveloped for American nurses could be used, after some smalladaptations, to obtain valid estimates of food intake in anAustralian adult population. We expanded a straightforwardevaluation of the questionnaire’s validity in this setting witha more in-depth investigation of the characteristics of par-ticipants that are associated with the differences between theFFQ and the reference method.

To our knowledge, there is no other Australian study thatpresented food-based validity of FFQ to which we can compareour results. Compared with food-based validation studies inother countries, the strength of the correlations obtained in thisstudy is comparable to studies in the United States (16), Mali(17), Guatemala (18), and Japan (19); with the exception of theUnited States (Nurses’ Health Study), all samples included bothmen and women. The correlation coefficients in our study werebetter than those reported for Chinese miners by Forman et al.(20), and were generally not as good as correlations observed inFrance (21) (nursing staff), Sweden (22) (population based),Finland (23) (pregnant women), and Shanghai/ China (24)(women). Importantly, the higher correlation coefficients weregenerally observed in single-sex studies. A correlation coefficientof 0.3 is a level at which attenuation is so severe that it would bedifficult to detect associations (25). The correlations #0.3 in ourstudy were for poultry, eggs, 5 vegetable groups, rice, pasta andnoodles, and nuts.

The ranking of individuals in terms of quartiles was assessedby comparing the quartile allocation of intake estimates fromFFQ with that of the WFR. Studies on diet and disease asso-ciations frequently divide food intake into quartiles. Overall, thepattern of results was similar to that of the correlations, but pooragreement and gross misclassification were not as extensive assuggested by the correlation results. Exact agreement in theallocation of food intakes by the FFQ and WFR ranged from 26to 63%, and agreement within 1 quartile ranged from 60 to 93%,results comparable to those reported in other food-based studies(17,19,24). The poorest agreement was observed for eggs,cruciferous vegetables, ‘‘other vegetables,’’ rice, and nuts, with

agreement for these food groups little different from thatexpected in random allocation. Median WFR intake estimatesgenerally showed increasing trends over increasing FFQ quar-tiles, although this was not monotonic for all food groups.

The poor agreement found for vegetables, assessed as a foodgroup or as individual foods, was also reported by studies in theUnited States (16), Guatemala (18), China (20) (miners), andFrance (21) (nursing staff), but not by others, i.e., Mali (17),Japan (19), Sweden (22) (population based), and Finland (23)(pregnant women). Nevertheless, Cade et al. (25) noted intheir review of validation studies for FFQ that mean corre-lations between FFQ and reference methods are usually lowestfor vegetables, explaining that misreporting of vegetables canoccur for a number of reasons including double counting ofitems and social desirability bias. For the other foods with pooragreement, the results from other studies are more varied. Forexample, studies have tended to report higher correlations foreggs than we observed, e.g., .0.4 in the United States (16) andGuatemala (18).

Agreement in the estimation of absolute intake was assessedin the limits of agreement analysis. Key findings were that onaverage the FFQ overestimated the intake of most foods, whichwas also reported in other validation studies (17,23,24), withlarge overestimations for intake of the fruit and vegetablegroups. This is reflected also at the individual level, with 4 ofthe 37 food groups having 95% lower limits #0.05, and 12having upper limits .20; there were very broad ranges for manyfoods, but a general trend for overestimation. The differencesbetween the FFQ and WFR intake estimates varied signifi-cantly with magnitude of intake estimates for 14 food groups,with 9 showing a negative association and 5 showing a positiveassociation. There was no clear pattern concerning which foodgroups had a negative, positive, or no association with mag-nitude of intake.

We reported previously that there were sex differences inthe extent of underreporting of energy intakes for FFQ andWFR in this study group (7). Using cut-off values based on theratio of energy intake to basal metabolic rate, as described byGoldberg et al. (26), the extent of underreporting was highestamong women using the WFR, ;2 times that observed forwomen using the FFQ, or for men using either method. In spiteof recognized weaknesses, the WFR is still regarded as themethod of choice to use as the reference method in validationstudies of this type. Bingham et al. (27) showed in anevaluation of 7 dietary assessment methods in comparisonwith several biomarkers of dietary intake that WFR wereconsistently more strongly associated with the biologicalmarkers than were the other methods. The authors concludedthat WFR ‘‘remain the most accurate measure of dietaryintake.’’ Cade et al. (28) in a recent review of FFQ also suggestthat WFR should be the first method of choice in validationstudies; a major advantage is that the main sources of errors aredifferent for the 2 methods, and unlikely to be correlated(correlated errors can lead to overestimation of validity by somemeasures). Thus, measurement errors in the WFR likelycontributed to the results observed in our study.

Nevertheless, the effect of sex on measures of agreement isillustrated with the observation above that FFQ validations insingle-sex studies tend to have higher correlations. It wassuggested that the FFQ format contributes to sex differences,and particularly that the treatment of portion sizes in datacollection contributes to both the sex differences observedbetween FFQ and WFR and the underreporting. Subar et al.(29) compared the validity in estimation of nutrient intakes for3 FFQ formats and found that the Willett instrument tends tounderestimate the nutrient intakes of men and overestimate

464 MARKS ET AL.

by guest on July 1, 2012jn.nutrition.org

Dow

nloaded from

those of women. They attribute it to the same portion sizesbeing assigned to men and women, as applied in this study. Ina recent review of the design, validation, and utilization of FFQ,Cade et al. (28) reported that in studies in which portion sizesare self-defined, there tended to be differences in portion sizebetween men and women, and further that correlations invalidity studies tended to be highest when subjects were able todescribe their own portion sizes.

The poor performance of vegetables across measures ofagreement is a matter of particular concern because of ourinterest in examining their potential role in cancer etiology. Forall vegetable groups, intakes were estimated by the FFQ to be;2 times those found with WFR. Correlations were modest,ranging from 0.08 to 0.40, and were generally stronger for fruits.This is consistent with other studies in which levels of agree-ment between the FFQ and other dietary assessment methodswere generally found to be poor for vegetables and fruits (30).Reasons for this are not well established.

The multivariable modeling in the limits of agreementanalysis shows that the models for fats and oils, poultry, seafood,various vegetable groups, rice, pasta and noodles, and all cerealsand products explained $25% of the variation in differencebetween FFQ and WFR. Sex was a significant explanatoryvariable for most of these. Of the other food groups with par-ticularly poor performance, the model explained ;10% of thevariation in difference.

These findings have important implications for modeling diet-disease relations. The significant association between personalcharacteristics and difference for most food groups raises thepossibility of differential bias and misclassification. Adjustingintake estimates for these characteristics will improve the validityof the model. One might expect this to be particularlyappropriate for the vegetable groups, for which the performanceof the FFQ is otherwise poor, and a reasonably large proportionof variation in difference is explained by the models. This wouldalso suggest that different subgroups of the study population mayneed different FFQ to accurately measure dietary intake; thisremains to be confirmed by further investigation.

This is the first study we know of that directly assessed theassociation between personal characteristics and measurementerrors in FFQ food intake estimates. It is widely acknowledgedthat a number of factors such as gender, age, and socioeco-nomic factors may be associated with the validity of dietaryestimates (3). Our study assessed the extent to which theyaffect measurement error in the Nambour study population. Ofall the personal characteristics studied, sex was most commonlyassociated with intake estimate errors for food groups; thepresence of a medical condition and dietary supplement intakewere also associated for some food groups. The findings high-light the need to assess FFQ validity in a sample that is rep-resentative of the overall population in which the FFQ will beused, with a sample size that is large enough to assess differencesamong subgroups.

ACKNOWLEDGMENTS

Bronwyn Ashton contributed significantly to early work related tothis study and Adrian Barnett was a discussant on methods of analysis.

LITERATURE CITED

1. Neuhouser ML, Patterson RE, Thornquist MD, Omenn GS, King IB,Goodman GE. Fruits and vegetables are associated with lower lung cancer riskonly in the placebo arm of the beta-carotene and retinol efficacy trial (CARET).Cancer Epidemiol Biomarkers Prev. 2003;12:350–8.

2. Block G, Hartman AM. Issues in reproducibility and validity of dietarystudies. Am J Clin Nutr. 1989;50:1133–8.

3. Nelson M. The validation of dietary assessment. In: Margetts BM, NelsonM, editors. Design concepts in nutritional epidemiology. 2nd ed. Oxford: OxfordUniversity Press; 1997

4. Marks GC, Hughes MC, van der Pols JC. The effect of personalcharacteristics on the validity of nutrient intake estimates using a food frequencyquestionnaire. Public Health Nutr. 2006; In press.

5. Green A, Williams G, Neale R, Hart V, Leslie D, Parsons P, Marks GC,Gaffney P, Battistutta D, et al. Daily sunscreen application and betacarotenesupplementation in prevention of basal-cell and squamous-cell carcinomas of theskin: a randomised controlled trial. Lancet. 1999;354:723–9.

6. Green A, Battistutta D, Hart V, Leslie D, Marks G, Williams G, Gaffney P,Parsons P, Hirst L, et al. The Nambour Skin Cancer and Actinic Eye DiseasePrevention Trial: design and baseline characteristics of participants. Control ClinTrials. 1994;15:512–22.

7. Ashton, B., Marks, G., Battistutta, D., Green, A. and The Nambour StudyGroup. Under-reporting of energy intake in two methods of dietary assessment inthe Nambour Trial. Aust J Nutr Diet. 1996;53: 53–60.

8. Willett WC, Sampson L, Stampfer MJ, Rosner B, Bain C, Witschi J,Hennekens CH, Speizer FE. Reproducibility and validity of a semiquantitative foodfrequency questionnaire. Am J Epidemiol. 1985;122:51–65.

9. Willett WC. Nutritional epidemiology. 2nd ed. New York:Oxford UniversityPress; 1998.

10. Radimer K, Harvey P, Green A, Orrell E. Compliance with dietary goals ina Queensland community. Aust J Public Health. 1992;16:277–81.

11. Department of Community Services and Health. National dietary survey ofadults: 1983. No. 2 Nutrient intakes. Canberra, Australia: Australian GovernmentPublishing Service; 1987.

12. Xyris Software Diet 1: nutrient calculation software. Version 3.1. HighgateHill, Queensland: Xyris Software (Australia) Pty Ltd; 1991.

13. Bland JM, Altman DG. Measuring agreement in method comparisonstudies. Stat Methods Med Res. 1999;8:135–60.

14. Ludbrook J. Statistical techniques for comparing measurers and methodsof measurement: a critical review. Clin Exp Pharmacol Physiol. 2002;29:527–36.

15. SAS Institute Inc. SAS system for Windows. Release 8.02. Cary, NC: SASInstitute; 1999–2001.

16. Salvini S, Hunter DJ, Sampson L, Stampfer MJ, Colditz GA, Rosner B,Willett WC. Food-based validation of a dietary questionnaire: the effects of week-to-week variation in food consumption. Int J Epidemiol. 1989;18:858–67.

17. Torheim LE, Barikmo I, Hatloy A, Diakite M, Solvoll K, Diarra MM, OshaugA. Validation of a quantitative food-frequency questionnaire for use in WesternMali. Public Health Nutr. 2001;4:1267–77.

18. Rodriguez MM, Mendez H, Torun B, Schroeder D, Stein AD. Validationof a semi-quantitative food-frequency questionnaire for use among adults inGuatemala. Public Health Nutr. 2002;5:691–9.

19. Ogawa K, Tsubono Y, Nishino Y, Watanabe Y, Ohkubo T, Watanabe T,Nakatsuka H, Takahashi N, Kawamura M, et al. Validation of a food-frequencyquestionnaire for cohort studies in rural Japan. Public Health Nutr. 2003;6:147–57.

20. Forman MR, Zhang J, Nebeling L, Yao SX, Slesinski MJ, Qiao YL, RossS, Keith S, Maher M, et al. Relative validity of a food frequency questionnaireamong tin miners in China: 1992/93 and 1995/96 diet validation studies. PublicHealth Nutr. 1999;2:301–15.

21. Bonifacj C, Gerber M, Scali J, Daures JP. Comparison of dietary assessmentmethods in a southern French population: use of weighed records, estimated-dietrecords and a food-frequency questionnaire. Eur J Clin Nutr. 1997;51:217–31.

22. Johansson I, Hallmans G, Wikman A, Biessy C, Riboli E, Kaaks R.Validation and calibration of food-frequency questionnaire measurements in theNorthern Sweden Health and Disease cohort. Public Health Nutr. 2002;5:487–96.

23. Erkkola M, Karppinen M, Javanainen J, Rasanen L, Knip M, Virtanen SM.Validity and reproducibility of a food frequency questionnaire for pregnant Finnishwomen. Am J Epidemiol. 2001;154:466–76.

24. Shu XO, Yang G, Jin F, Liu D, Kushi L, Wen W, Gao YT, Zheng W. Validityand reproducibility of the food frequency questionnaire used in the ShanghaiWomen’s Health Study. Eur J Clin Nutr. 2004;58:17–23.

25. Cade J, Thompson R, Burley V, Warm D. Development, validation andutilisation of food-frequency questionnaires—a review. Public Health Nutr. 2002;5:567–87.

26. Goldberg GR, Black AE, Jebb SA, Cole TJ, Murgatroyd PR, Coward WA,Prentice AM. Critical evaluation of energy intake data using fundamental principlesof energy physiology: 1. Derivation of cut-off limits to identify under-recording. Eur JClin Nutr. 1991;45:569–81.

27. Bingham SA, Cassidy A, Cole TJ, Welch A, Runswick SA, Black AE,Thurnham D, Bates C, Khaw KT, et al. Validation of weighed records and othermethods of dietary assessment using the 24 h urine nitrogen technique and otherbiological markers. Br J Nutr. 1995;73:531–50.

28. Cade JE, Burley VJ, Warm DL, Thompson RL, Margetts BM. Food-frequency questionnaires: a review of their design, validation and utilisation. NutrRes Rev. 2004;17:5–22.

29. Subar AF, Thompson FE, Kipnis V, Midthune D, Hurwitz P, McNutt S,McIntosh A, Rosenfeld S. Comparative validation of the Block, Willett, and NationalCancer Institute food frequency questionnaires: the Eating at America’s TableStudy. Am J Epidemiol. 2001;154:1089–99.

30. Michels KB, Welch AA, Luben R, Bingham SA, Day NE. Measurementof fruit and vegetable consumption with diet questionnaires and implications foranalyses and interpretation. Am J Epidemiol. 2005;161:987–94.

465FACTORS ASSOCIATED WITH VALIDITY OF FFQ FOOD ESTIMATES

by guest on July 1, 2012jn.nutrition.org

Dow

nloaded from