Validity Oral

8/2/2019 Validity Oral

1/9

206 Journalof Public Health Dentistry

Validity of Two Methods for Assessing Oral Health Statusof PopulationsEugenio D. Beltran, DMD, MPH, MS, DrPH; Dolores M. Malvitz, RDH, MPH, DrPH;Stephen A. Eklund, DDS, MHSA, DrPH

_ _ - _ _ ~ -bstractObjective: This investigation assessed two methods for estimatingepidemiologic indicators of oral health status among children: (1 ) a visual-onlyscreening,performed independently by a dental hygienist and a registered nurse;and (2) a parent- or guardian-completed questionnaire. The indicators includeddichotomous variables measuring dental caries and treatment needs, presenceof sealants, njuries to the anterior teeth, and dental fluorosis.Methods:Followingtraining and calibration, data were collected over an eight-day period in April 1994among 632 elementary schoolchildren (aged5 to 1Pyears) n Monticello, Georgia.Both screening and questionnaire findings were compared pairwise with resultsfrom visual-tactile examinations done by a dentist. Validity, represented bysensitivity, specificity, and predictive values, was assessed for screening resultsfrom the dental hygienist, the nurse, and the parent-completed questionnaire.Results: Validity was high for screening for caries and treatment needs (>go%for sensitivity, specificity, and predictive values in a sample having 30% to 40%prevalence). Less valid data -mainly an effect of false negatives- ereobtained for fluorosis, njuries, and presence of sealants. No significant differencein validity was observed between the nurse and the dental hygienist. One-third ofrespondents to the questionnaire did not know if their children needed fillings (aproxy for untreated decay) or had received sealants; only knowledge of restora-tions was comparable to results from screening. lntraexaminer reliability for thetwo screeners ranged from 85 to 100 for percent agreement and 0.70 to 0.93 forkappa scores. Conclusions: Screening by dental hygienists or nurses canprovide valid data for surveillance of dental caries and treatment needs. Trainingfor visual assessment of fluorosis and injuries must be improved to diminish theproportion of false negatives. A parent-completed questionnaire is less effectivethan visual screening for evaluating oral health status in children. [J Public HealthDent 1997;57(4):206-2141

~ ~

Key Words: validity, reliability, visual-tactile examination, screenings, question-naire, oral health assessment, surveillance.

Surveillance in public health is de-fined as the ongoing systematic collec-tion, analysis, and interpretation ofoutcome-specific data for use in theplanning, implementation, and evalu-ation of public health practice (1).Sev-eral approaches and sources of dataare used in public health surveillance:vital statistics, notifiable disease data,registries, sample surveys, adminis-trative data, and sentinel surveillancedata (2) .An important element of any

surveillance system is the use of thedata to advance public health.No true surveillance systems fororal conditions exist. The prevalenceand trends of oral diseases have beenmonitored through oral health sur-veys at the national and sometimesstate and local levels. These surveyshave used some sort of clinical exami-nation and questionnaire. A dentalprofessional generally performs a vis-ual-tactile assessment of the mouth in

a sample of the population. Many ofthese surveys have used the Radikediagnostic criteria and examinationprotocol (3), modifications of theRadike criteria such as those by theNational Institute of Dental Research(4),or those of the World Health Or-ganization (5).Some differences exist between thediagnostic criteria used in these sur-veys and those used by clinicians intheir practices. In clinical examina-tions practitioners make complexmeasurements, stressing the precisionof the process, i.e., the detection of themost incipient signs of disease. Thegreater the complexity, however, thegreater the likelihood of unreliable re-sults. Research shows that cliniciansapply different criteria (6,7)and differsignificantly in their diagnoses andtreatmentplans (8-11).Oral health sur-veys have emphasized the reliabilityof measurements. Experience sug-gests that visual-tactile examinationscan be used to estimate the oral healthsta tus of populations, provided the as-sessment is performed in well-definedsamples by trained examiners whousevalid indices with known reliability. Inoral epidemiology these surveys areconsidered the standard, even thoughsome underdiagnosis is expected dueto the lack of procedures and tech-niques that are feasibleor practical forfield examinations (e.g., x-rays, opticaltransillumination). Still, oral healthsurveys are complex and demand ex-tensive resources, which might ex-plain why many programs lack dataon oral health status (12).Screeningsare defined in the medi-cal model as procedures that can sortout persons who may have a conditionfrom those who may not (13). Thosewho appear to have the condition are

- _ __ _ ~ - _ _ ____-__. - . -Send correspondence and reprint requests to Dr. eltran, Divisio n of Oral Health, Centers for Disease Control and Pre vention, NCC DPH P, 4770Buford Highway , MS F-10, Chamblee, GA 30341. E-mail: edM@ cdc.gov.Dr. Malvitz is chief, Surveillance, Investigations, a nd Kesearch Branch,Bv i s i on of Oral H ealth, CDC. Dr. Eklund is associate professor, School of Public Health, University of Michigan, AM Arbor. llus manusaipt waspresented as a poster a t the annual meeting of the American Association for Dental Research in San Francisco, April 1996. Man uscript recvived:8/26/96; returned to authors for revision: 10/29/96; accepted for publication: 4/18/97.


2/9

Vol. 37, No.4, Fall 1997 207followed up to obtain a final diagnosis(13).Because screenings are applied topopulations, their methods must besimple, inexpensive, and require mini-mal training for application and inter-pretation, e.g., the PPD tuberculin test.Screenings must be evaluated againstsome standard procedure for valid-ity-that is, their sensitivity, specific-ity, and predictive value. In this evalu-ation, the proportion of false positivesand false negatives are important topatient follow-up and cost (14).Frankenburg (15) has provided thefollowing 10 criteria for selecting dis-eases or conditions for screening pro-grams: ( I ) the condition has to be seri-ous or potentially so, (2) i t should bepossible to differentiate those with thecondition from those at borderline orwithout it, (3) the prognosis should beimproved if the condition is detectedand treated early, (4) there should beadequat e lead and screening time, (5)the condition should be treatable orcontrollable, (6) the condition shouldbe prevalent, (7) there should be noharm caused to the individual beingscreened, (8) here should b e a follow-u p for those found positive, (9) theprocedure should be cost effective,and (10) the program should be ac-ceptable to the public.Screening protocols for oral canceror precancerous lesions reported inthe literature (16-19) involve the sys-tematic and consistent visual-tactileexamination of all anatomical sites ofthe oral and pharyngeal cavity andneck (20). Few oral conditions, otherthan oral cancer, have been the subjectof screening programs and activities(21-23). For den tal caries, screeningdenotes a fast and simple process bywhich an examiner visually inspectsthe oral cavity to detect the presenceor absence of specific oral conditions(24),as has been done in health fairs(25). Based on Frankenburgs criteria,dental caries, dental fluorosis, anddental injuries screening protocolscould be developed as alternatives tovisual-tactile examinations.Few studies have tested the visualand visual-tactile abilities of dentaland nondental personnel other thandentists. Only in a few large-scale sur-veys have dental hygienists been ex-aminers (26-30). These studies re-ported interexaminer reliability tocompare the examinations made byhygienists and dentists. None of thesestudies evaluated validity. Two stud-

ies from the s ame research team(29,301 reported measures of validity;however, these measures were on theability of their research instrument (amultivariable statistical model) to pre-dict future caries patterns. Basedmostly on reliability values, thesestud ies concluded that dental hygien-ists could be used to collect oral healthstatus data. A few studies (2531) havefocused on the use of other person-nel-such as denta l auxiliaries, dentalstuden ts, or teachers-for examina-tions or screenings; however, none ofthese studies have assessed validity.Oral health status and behaviorsalso have been assessed by question-naires, for example those of the Na-t ional Health In terview Survey(NHIS) (32) and the Behavioral RiskFactor Surveillance System (BRFSS)(33). These data a re self-reported andsubject to error, particularly selectionand information biases (34,35). Severalstudies have compared data obtainedfrom clinical examinations with thoseself-administered questionnaires (36-411 phone interviews (421, and parent-responded questionnaires (43). De-spite the different methodologies,populations, oral conditions assessed,and the fact that many studies evalu-ated reliability instead of validity, re-searchers concluded that question-naires appear to be useful tools forobtaining oral health data.

The objective of this investigationwas to test the validity of visualscreening (protocol 1) and parent- orguardian-completed questionnaire(protocol 2) for assessing the oralhealth sta tus of schoolchildren, withthe goal of using these protocols astools in a surveillance system for oralconditions. The visual-tactile exami-nation was used as the standard pro-tocol.Methods

Children in kindergarten throughfifth grade (N=838;ged 5 to 12 years)in a rural school in Monticello, Geor-gia, were invited to participate in thisinvestigation. This school had askedthe Georgia Department of HumanResources to evaluate the childrensdental status. One week before exami-nation, children were given a packagethat included a letter of invitation, aconsent form, and a multiple-choicequestionnaire for their parents. Allchildren who returned questionnairespart icipated in the examination

(n=632; 75% response rate). Followingexamination, a form with recommen-dations regarding the childs need fordental treatment was sent to the par-ents.All clinical procedures followedguidelines for infection control in den-tal settings (44,45). A protocol for in-jury exposure control consistent withCDC guidelines and the OccupationalSafety and Health Administrationstandards was followed. Institutionalreview board approval was obtainedfrom both CDC and the University ofMichigan.

Measurement Protocols. Th estandard examination protocol was avisual-tactile examination of all toothsurfaces in each child. All childrenwere examined by a dentist who hadbeen calibrated using NIDR diagnos-tic criteria for dental caries and pres-ence of sealants (4), Deans index fordental fluorosis (41, and Bhats criteriafo rdental injuries (46).A portable den-tal chair and light, dental explorers,and plane-surface mirrors were used;no radiographs were exposed. Fromthese examinations, each child wasclassified into one of the followinggroups based on dental caries in theirprimary and permanent teeth: (1)car-ies free; (2) at least one tooth restored,but no untreated decay; or (3) at leastone untreated carious lesion, whetherrestorations were present or not. Inaddition, each child was classified ashaving (yes/no): (1) at least one per-manent molar with pit and fissuresealants, (2) at least one anterior toothwith injury or sequelae, and (3) dentalfluorosis. Finally, each child was clas-sified into a three-level variable ac-cording to urgency of treatmentneeded (no need, nonurgent, and ur-gent). Need for urgent treatment wasdefined a s having one or more teethwith extensive tissue destruction,probably involving the pulp, or his-tory of pain or signs of infection.Nonurgent needs included presenceof untreated carious lesions or calcu-lus.Two measurement protocols weretested. Protocol 1 was a visual-onlyscreening of the childsmouth done bya dental hygienist or a registerednurse, each of whom screened aboutone-half of each class in sequential or-der. The dental hygienist had not per-formed oral assessments or providedtreatment for more than five years.The nurse had no previous clinical


3/9

208 Journal of Public Health Dentistrydental experience. Both screeners re-ceived written material on the proce-dures and diagnostic criteria for eachcondition to be evaluated.The protocol allowed the dental hy-gienist to perform oral screenings afterreview of this material and withoutfurther training. A five-hour trainingsession, which used both slides andpatients, familiarized the nurse withdental nomenclature and clinical fea-tures of each condition. The protocolrequired the screener to inspect allfour quadrants of the mouth sequen-tially and to code the results accordingto fivealgorithms. Figure 1depicts thealgorithm used to code dental caries (afull set of these algorithms is availableupon request). In the particular case ofcoding for dental caries, some childrenwith restorations were coded as 2because they also had untreated cari-ous lesions. Screenings were carriedout using a portable chair, a flashlightfor intraoral i l lumination, and atongue blade to facilitate visual accessduring inspection. Visual screening al-ways preceded the visual-tactile ex-amination for each child. Approxi-mately 5 percent of the children werereexamined by the dentist and the twoscreeners to estimate intraexaminerreliability. Most replicate assessmentswere performed 2C-25 minutes after

the initial dental screenings and 45-60minutes after the initial visual-tactileexaminations.Protocol 2 was a 17-item, multiple-choice questionnaire completed by aparent or guardian when permissionto participate wa s granted. The q u e s

tionnaire asked about conditions pre-sent in the childs mouth, demo-graphic characteristics, and family so-cioeconomic status. Questions aboutdental fluorosis and need for dentaltreatment were not included; dontknow options were available. The

FIGURE 1Visual Screening Algorithm for Coding Dental Caries

Do a cycle through the four quadrants, then answer:IEeIs here any dk ay ed ooth?]--+ N OatDo another cycle and answer:t

I 1IThen code caries [2] I Then code caries [l] LThen code caries [O]TABLE 1

Validity of Visual Screening Versus Visual-tactile Examination in Evaluating Presence of Selected Oral Conditions,Both Screeners Combined

Condition N* Sensitivity (%)Caries3Restorations present

Untreated decayCaries experiences

FluorosisInjuriesSealantsTreatment7Nonurgent

UrgentAny treatmentneeded

421434632632632632554482632

94.093.694.378.680.059.173.398.483.4

Predictive Value (%)Specificity(%) Positive Negative

99.6 99.4 %.296.9 95.3 95.896.6 97.5 92.395.9 90.9 89.798.2 85.7 97.399.7 92.9 97.095.7 81.7 93.199.8 98.4 99.795.4 89.0 92.9

Prevalence (%)t

39.739.958.534.011.97.020.912.930.5

Total samplesize available (includingthe reference group) o assess validity for each condition.+A s determmed by the msual-tactile examination.6Caries experiencewas detined a s having treated or unaeated cariouslesions. N o need for treatment was used as the reference.Caries free was used as the reference.


4/9

Vol. 57, No.4, Fall 1997 209TABLE 2

Validity of Visual Screening by Dental Hygienist Versus Visual-tactile Examination in Evaluating Presenceof SelectedOral ConditionsPredictive Value (% )

ConditionCaries$

_ _

Restorations presentUntreated decayCaries experiences

Fluorosis1n juriesSeaIan tsTreatmen 91Nonurgen

UrgentA ny treatmentneeded

w~

188215309309309309266221309

Sensitivity(%)

98.094.893.788.280.659.179.497.186.7

Specificity (%)-~~.100.0

94.194.195.598.999.791.699.591.2

Positive

100.092.996.288.290.692.974.697.183.5

Negative_ _94.195.790.395.597.597.093.599.593.0

Prevalence(70)t~

40.444.761.527.511.77.1

23.715.434.0

Total sample size available (including the reference group) to assess validity for each condition.tA s determined by the visual-tactile examination.fCaries experience was defined a s having treated or untreated carious lesions. N o need for treatment was used a s t h e reference.

Caries free was used as th e reference.

questionnaire was reviewed andtested by persons with expertise inquestionnaire development and by aconvenience sample of paren ts andschool eachers.Data Management and StatisticalMethods. Data were entered directlyinto computer files using customizeddata entry programs in the Epi Infosoftware (47). Following dat a cleaningand editing, epidemiologic indicatorswere calculated using SAS software.Later, the screening and the question-naire data sets were matched withdata from the standard protocol usingall eligible records to maximize sam-ple size. Pairwise matching of thescreening data with the standard pro-tocol produced 3x3 contingency tablesfo r dental caries and treatment ur-gency and 2x 2 tables for all other con-ditions. Pairwise matching of ques-tionnaire data with the standard pro-tocol produced 2x2 tables. To assessthe validity of each alternative proto-col in each condition, a 2x2 table wasprepared; for dental caries and treat-ment urgency in the screening proto-col, only the appropria te cells from the3x3 table were included.All measures of validity- ensitiv-ity, specificity, positive predictivevalue, and negative predictive value-and the prevalence of the condition inthe sample were calculated using

standard procedures (48). When ap-propriate, standard errors and confi-dence intervals were estimated usingbinomial approximation. Data werestratified to control for sociode-mographic variables. Data from dupl i-cate examinations and screeningswere used to check for intraexaminerreliability tested by percent agreementand kappa statistics (49).Results .Validity of Visual Screenings andType of Screener.Summaries of valid-ity results for screening are includedin Tables 1-3. In each table two addi-tional comparisons grouped thosewith a ny caries experience (treatedoruntre ated ) and those needing anytreatment. Tables 1 4 isplay the totalnumber of subjects (N) used in the sta-tistical procedures to assess validityfor each condition, including those inthe reference gro up (those not havingthe condition).

Specificity for both screeners com-bined (Table 1) was greater than 95percent across all oral conditions; the95 percent confidence intervals variedfrom 94 to 100 percent (data not dis-played). The sensitivity for caries wasaround 94 percent (95%CI=91%,96%).The lowest sensitivity was found inscreening for sealants (59%; 95%CI=55%, 63%), followed by nonurgent

treatment needed (73%), fluorosis(79%), and injuries (80%).A positivepredictive value of 89 percent orhigher was found for most oral condi-tions; exceptions were nonurgenttreatment needed (82%) and injuries(86%). Negative predictive valueswere 90 percent or higher across allconditions, and many values wereover 95 percent.The dental hygienist (Table 2 )screened 309 children with moreprevalent untreated decay than in the323children screened by thenurse; thesample screened by the nurse had ahigher prevalence of fluorosis (Table3). Results obtained by the dental hy-gienist and nurse were comparable tothose obtained by the dentist (confi-dence intervals overlapped); never-theless, some minor differences werefound. For example, the dental hy-gienist performed slightly better thanthe nurse in assessing fluorosis anddetermining the need for nonurgentand any treatment (Tables 2 and 3).These differences could be explained,in part, by the differences in preva-lence in some of these conditions. Theoverall lower sensitivity for nonurgenttreatment needs may be a conse-quence of the lower prevalence ofthese conditions in children screenedby the nurse. The lack of a tactile com-ponent in the assessments made by


5/9

210 Journal of Public Health DentistryTABLE3

Validity of Visual Screening by Nurse Versus Visual-tactile Examinationin Evaluating Presence of Selected OralConditions

ConditionCaries$~- ~~ ~ -Restorations presentUntreated decayCaries experiences

FluorosisInjuriesSealan tsTreatment

NonurgentUrgentA n y treatmentneeded

w_ _233219323323323323288261323

Sensitivity (% ) Specificity(9%)~ ~ -~ -~ -

96.7 99.392.2 99.395.0 98.672.3 96.479.5 97.559.1 99.766.0 99.2

100.0 100.079.6 99.2

Predictive Value (5%)Positive-~ ~-

98.998.698.893.181.692.994.6

100.097.2

Negative__ -97.995.994.083.897.297.192.8

100.092.8

39.135.255.740.312.16.8

18.410.727.2

Total samplesize available (including the reference group) to assess validity for each condition.t As determined by th e visual-tactile examin ation.iCaries experiencewas defined as having treated or untreated carious lesions. N o need for treatment was used as the reference.Caries free was used as the reference.

TABLE4Validity of ParenVGuardian-completed Questionnaire Versus Visual-tactile Examination in Evaluating Presence

of Selected Oral ConditionsPredictive Value ( 7 0 )Excluded n Sensitivity Specificity Prevalence

N ( 7 c ) (?GI (5%) Positive Negative (70)_ _ _ _ _ ~ _ _ ~ ~ _ _ _ _ ~ _ _ _ _~~ ~~~ . ___ ~~~N o caries vs restorations 401 60 (10) 93.3 89.1 84.5 95.1 40.7N o caries vs untreated 305 214 (34) 68.8 88.3 80.4 80.3 40.9Injuries 611 17 (3) 20.0 87.3 16.9 89.4 11.5Sealants 423 205 (33) 56.7 89.3 28.8 93.4 7.1

prescntdecay

As determined by visual-tactileexamination

both screeners may be the cause of thelower sensitivity for sealants.

Validity of Questionnaires. Aboutone-third of respondents did notknow or did not indicate whether theirchildren had received sealants orneeded restorative treatment. A lowerproportion (10%)did not know if theirchildren had restored teeth. To avoidclassification bias, these observationswere not included in the assessment ofthe questionnaires validity (Table 4).The best results were obtained inevaluating the presence of restorations(93%sensitivity and95%negative pre-dictive value). Although these values

were similar to those obtained byscreening (Table 11, specificity andpositive predictive value were lowerin the questionnaire (89% and 85%,respectively) than in the screening.Measures of validity were lower forthe other three conditions (untreateddecay, injuries, and presence of seal-ants), as well.To evaluate any potential differ-ences among respondents ability tocorrectly assess the presence of resto-rations, responses were stratified byfamily income (2$30,000/year vsother), eligibility for free or reduced-price lunch, number of children in the

family (23vs other), and respondentslevel of education (>high school vsother). Respondents with greater fam-il y income or having children not eli-gible for free or reduced-price lunchassessed presence of restorations intheir children with higher sensitivity(data not shown).

Reliability of Measurements. Per-cent agreement and kappa estimatesfor each condition were calculatedseparately for the dentist, the dentalhygienist, and the nurse (Table5).Thedentist evaluated caries status andpresence of sealants on each surface byusing 14 mutually exclusive diagnos-


6/9

Vol. 57 , No.4, Fall 1997 211TABLE 5Intraexaminer Reliability for Examiner and Screeners

Caries+ Sealantst Treatment Urgency$ Fluorosis ~njuriesy% Agree. Kappa % Agree. Kappa ?& Agree. Kappa YoAgree. Kappa YOAgree. Kappa

Dentist 99 0.97 100 1 o 98 0.92 99 0.92Hygienist 85 0.77 100 1.00 89 0.77 96 0.92 96 0.70Nurse 96 0.93 100 1 oo 100 1 .o 91 0.81 96 0.83The dentist evaluated canes status on each surface by using 14 mutually exclusive diagnostic categories; both sue ene rs evaluated caries in theentire mouth by using three possible diagnostic categories.tThe dentist evaluated presence of sealants as part of the magnosis of canes.iBased on two diagnostic categories.YThe denhst evaluated injuries by using four &agnostic categories for each of the upper front teeth; both screene rs evaluated injuries in the upperanterior teeth by using two &agnostic categories

Based on three &agnostic categories.

tic categories; therefore, presence ofsealants was not analyzed a s an inde-pendent item for the dentist. The den -tists reliability was evaluated over 14diagnostic categories for caries andpresence of sealants, seven for pres-ence of injuries (46), three for treat-ment urgency, and two for dentalfluorosis. Both the dental hygienistand nurse were evaluated using three-level variables for caries and treatmenturgency and two-level variables forpresence of sealants, dental fluorosis,and dental injuries. All examinersshowed high reliability: agreementswere greater than 85%, and kappa sta-tistics were greater than 0.70, a sub-stantial and almost perfect score ac-cording to the scale proposed by Lan-dis and Koch (50).Discussion

Validity of Visual Screening andType of Screener. Visual screeningscan produce data highly comparableto those obtained from visual-tactileexaminations. The highest validitywas found in the screening for canesand urgent treatment (Table 1) in asample for which the prevalence ofdental caries was comparable to theentire population of Georgia (51)andthe United States (4). The protocol forscreening did not discriminate be-tween dental caries of the primary andpermanent dentitions because its ob-jective was to obtain epidemiologic in-formation on the overall caries experi-ence of the subject. If necessary, theprotocol could be modified to measureeach dentition separately.With the exception of negative pre-dictive values for caries experience,predictive values were greater than 95

percent, which indicates that if the vis-ual screening protocol were applied toa sample having a similar caries preva-lence, at least 95 ou t of every 100posi-tive and negative cases would be truepositives and true negatives. These re-sults suggest that the visual screeningprotocol could provide valid epide-miologic information for dental cariesand restorations.The lower validity of screening fortreatment needs in this investigationmay indicate uncertainty in translat-ing the diagnosis of untreated decayinto the category of nonurgent treat-ment. A solution would be the use ofa computer algorithm incorporatedinto the data entry program thatwould automatically assign a code fortreatment urgency by taking into ac-count the information entered for car-ies and responses to yes or no ques-tions about other oral health variables.

Lower validity also was found forother oral conditions (fluorosis, inju-ries, and presence of sealants). Injuriesand presence of sealants were not asprevalent a s dental canes in the sam-ple, which might explain some of theseresults. The low sensitivity for pres-ence of sealants (59%)was probably aneffect of the lack of tactile reference.Although fluorosis wasas prevalent ascaries in the sample, a significantnumber of false negatives (46 out of215) were responsible for the lowersensitivity (79%)and negative predic-tive value (90%).

Some researchers have suggestedthat validity could be enhanced by im-proving the diagnostic criteria, re-training, and follow-up of training(16,521. Diagnostic criteria and codingin this investigation were designed to

be simple to understand and apply.The validity of screening for injuriesand fluorosis could be improved byincreasing the length and quality oftraining and allowing exposure to awider variety of clinical cases. How-ever, improved training may not affectthe validity of screening for dentalsealants d ue to the lack of tactile refer-ence.Few screening protocols for oralconditions have been proposed andtested; only screening programs fororal mucosal lesions have been testedfor validity (16,17,19).Some investiga-tors used findings from these screen-ing protocols to estimate epidemi-olo g~c arameters (53-55).Screeningprograms for other oral conditions, es-pecially dental caries, have been im-plemented without testing their valid-ity (25). Anecdotal information sug-gests that screening protocols, such asthe one in the Association of State andTerritorial Dental Directors seven-step model (24), are being increas-ingly used, in part because of the fi-nancial and logistic difficulties associ-ated with visual-tactile examinations.The current investigation also fo-cused on potential differences in thevalidity of measurements obtained bya registered nurseand a dental hygien-ist. Except for sensitivity for fluorosisand nonurgent treatment, the nurseprovided information as valid as thatprovided by the dental hygienist. Inboth cases, this finding could be ex-plained in part by differences in preva-lence. The hygienists higher sensitiv-ity and lower specificity values for un-treated decay (95% and 94%,respectively) could be explained alsoby a higher prevalence in her sample


7/9

212 Journal of Public Health Dentistry

(45%). However, the hygienist had asubstantially larger proportion of falsepositives, with a direct effect on herpositive predictive value (93%).Possi-bly, t h e hygienist's previous clinicaltraining, focused on evaluating incipi-ent signs of disease, is responsible forthe large proportion of false positives,a tendency that could have been ag-gravated by the absence of a confirma-tory tactile reference.Few studies have reported on thequality of the data collected by person-nel other than dentists in performingvisual or visual-tactile intraoral exami-nations (26,29). Direct comparisonwith their results is not possible be-cause of their focus on reliability or useof different methodologies. However,we agree that personnel other thandentists can be trained to perform oralhealth status assessments. Primaryhealth care workers have been used inSri Lanka and India to perform screen-ings to detect oral mucosal lesions(1656). Observations by nondentistscan be highly valid if the criteria areclear and appropriate training is pro-vided. In this investigation we chose aregistered nurse because of her avail-ability. Her normal duties include thehealth assessment of children in Mon-ticello, and we foresee theseepidemiologic assessments as part ofroutine health examinations. What isparticularly remarkable from the pre-sent findings is that a nurse, who hadlittle formal dental training, was ableto use the screening criteria and ade-quately apply them.

Validity of Questionnaires. Ques-tionnaires completed by parents orguardians were tested for their valid-it y in obtaining status information ona selected group of oral conditions, i.e.,presence of restorations, untreated de-cay, sealants, and injuries to the ante-rior teeth. The instrument included a"don't know" option in all questionsto avoid forcing the parent to "guess"the status of their children. A fairlyhigh proportion of respondents didnot know the oral health sta tus of theirchildren. Validity was modest for allfour indicators except for restorations(Table 41, suggesting limitations fortheir use in surveillance.

Many studies have reported on t h equality of self-reported information ondifferent aspects of oral health. A se-ries of studies published in the 197Os,1980s, and early 1990s in Scandinavia(36-41 , England and Ireland (43,57),

and the United States (42) nvestigatedthe quality of data collected throughquestionnaires. Data collected in theseinvestigations included the respond-ers' number of teeth and use of den-tures, presence of caries and gingivaldiseases, and personal medical his-tory. These investigations comparedthe self-appraised oral health statuswith results from a clinical examina-tion. These studies tested reliability(sometimes labeled as "agreement" or"validity") by using various statistics,including percent agreement, kappa,statistical mean, and Pearson's prod-uct moment correlation. Some studiesdescribed overreporting, e.g., havingmore tee th than those present(36,41,43), others underreporting(38,40),or both (37). This inconclusiveevidence suggests that the reliabilityof self- or proxy-reported oral healthdata is limited.In this investigation, analysis by so-ciodemographic variables showed noclear or consistent difference from theoverall results. This finding may beexplained, in part, by the sociode-mographic homogeneity of the chil-dren in the sample.

How "Good" is a "Valid" Proce-dure? A review of publications in thedental literature reveals subjective as-sessments by investigators on howgood they view the quality of theirresults based on values for validity.For example, for O'Sullivan and Ti-nanoff (58),a positive predictive valueof 87 percent for canes in the maxillaryteeth a s an indicator of caries risk inthe pits and fissures of posterior teethin the primary dentition one year laterwas acceptable. Bretz and co-workers(59) concluded that PERIOSCANTM,which showed 91 percent sensitivityand 89 percent specificity, was a validinstrument for detecting bacterfalcolonization of root surfaces. Pietilaand associates (22)accepted specificityvalues between 72 percent and 92 per-cent as valid. Ikeda and associates (17)concluded that a positive predictivevalue of 71 percent was fairly good.Warnakulasuriya and Pindborg (16)concluded that a positive predictivevalue of 58 percent was successful. Ina recent publication, Brunette (60)pre-sented a sample of sensitivity andspecificity values for diagnostic testsused in dentistry taken from the litera-ture. Sensitivity for dental can es wasas low as 13percent and as high as 93percent. Clearly, a wide range of ac-

ceptance levels exists. In fact, few stud-ies have concluded that the proce-dures being tested were not valid. Thisconclusion can mislead potential userswho might apply a procedure withborderline validity in a low prevalencesample and reach incorrect conclu-sions (61).The problem of defining how"good" a procedure is based on a va-lidity result depe nds on many aspects,including the nature and quality of thestandard used, the prevalence of thecondition being assessed (maximizingpositive predictive value with mini-mum effect on negative predictivevalue), and the proportion of falsepositives and false negatives the userof the procedure is willing to accept.These arguments reduce the problemfrom how "valid" a procedure is to"how practical" it is for its intendedpurpose. In fact, oral epidemiologiststrade off val idity for reliability inmany indices and procedures. Giventhis circumstance, the screening pro-cedure tested in this investigation ap-pears useful for estimating dental car-ies in samples having a similar preva-lence to this study.Surveillance systems for oral condi-tions are so underdeveloped that al-most every aspect of these systems re-quires empirical support. The need fororal health status and treatment needsdata is increasing because dental pub-lic health programs must support theirneed for and use of resources. Thisinvestigation has tested a visualscreening protocol suitable as a sur -veillance tool for collecting oral healthstatus data. Parent- or guardian com-pleted questionnaires seem to be oflimited us e mainly because a high pro-portion of parents did not know theirchildren's oral health status, otherthan the presence of one or more res-torations. Because questionnaires areunlikely to be designed to addressonly one topic, if necessary, this ques-tion on restorations could be includedin ongoing structured questionnaireswith a wider range of topics.References

1. ThackerSB,Berkelman RL. Public healthsurveillance in the United States.Epidemiol Rev 1988;1016490.2.Stroup NE, Zack MM, Wharton M.Sources of routinely collected data forsurveillance. In: Teutxh SM , ChurchillRE, eds. Principles and practice of publichealth surveillance. New York: OxfordUniversity Press, 1994:31-85.3. Radlke AW. Criteria for diagnosisof den-


8/9

Vol. 57, No.4, Fall 1997tal caries. Proceedings of the conferenceon the c l in ical tes t ing of cariostaticagen ts. Chicago: American Dental Ass@aat ion , 1968.4. National Institute of Dental Research.O r a l health of United Sta tes children.Washington, DC: U S Public Health Serv-ice, 1989; pu b n o (NIH) 89-2247.5. World I lealth Organization. Oral he althsurvey s basic methods. 3rd ed. Geneva:WllO, 1987.6. Riordan PJ, Espelid I, Tveit AB. Radio-graphic interpretation an d treatment de-as io ns among den tal therap is ts and den-tists in Western Australia. CommunityDent Oral E pidem iol 1991;19:268-71.7. Knutsson K, Brehmer B, Lysell L, RohlinM. General dental practitionersevalu-ation of the need for extraction of a s y m ptomahc mandb u lar th i rd molars . Com-munity Dent Oral Epidemiol 1992;20:347-50.8. Rytomaa I, Jarvinen V, Jarvinen J. Vari-a ti on in c a n e r eco rd n g an d r e sto ra t iv etreatment plan am ong university teach-ers. Community Den t Oral Ep idemio l1979;7335-9.9. Mileman P, Purdell-Lewis D, van derWeek LT. Variation in treatment dea-s ions and rad ograp h ic car ies d iagnosisamong un ivers ity teachers . CommunityDent Oral Epidemiol1982;10329-34.10. Bader JD , Shugars DA. Unders tand ingdentists restorative treatment decisions.J Public I lealth Dent 1992;52:102-10.11. B a d e r JD , S h u g a r s D A . A g r e e m e n tamong den t is ts recommendat ions fo rrestorative treatment. J Dent Res 1993;72:

12. Centers for Disease Control and Preven-tion. Core public health functions andstate efforts to improve oral health-United States, 1993. MMW R Mo rb Mor-tal Wkly Rep 1994;43:201,207-9.13 . Commission on Ch ronicIllness.Chronicillness in the Unite d States . Vol. 1. Pre-vention of chronic Illness. Cam bridge,MA : f farvard University Press, 95745 .14. Morrison AS. Screening in chronic d isease . N ew Y o rk : O x fo rd U n iv e r s i t yPress, 1985.15. Frankenburg WK. Pnnciples in selectingd i s e a s e s f o r s c r e e n i n g - c r it e r i a i nscreening test selection. ln:FrankenburgWK, Cam p BW, eds . Pediatric screening

891-6.

tests. Sprhgf ie ld , IL : Char les C . Thom ag1975:9-37.16 . WarnakulasuriyaS, PlndborgJJ. Reliabil-it y of oral precancer screening by pri-mary health care workers in Sri Lanka.Com mun ity Dent H ealth 1990;772-9.17. Ikeda N, lshii T, Iida S, Kawai T. Epide-mio log ical s tudy of o ra l l eu k o p lak iabased on m ass screening for oral mucosaldseases in a selected Japanese popula-tion. Community Dent Oral Epidemiol1991;19:1M)-3.18. B a n k z y 1, Rgo 0.Prevalence stu dy oforal precancerous lesions within a com-plex screening system in Hu ngary . Com-munity Dent Oral Epidemiol 1991;19:265-7.19. Downer MC, Evans AW, Hug hes HalletCM, Jullien JA, Speig ht PM, Zakrzew skaJM. Evaluation of screening for oral can-ce r and precancer in a company head-quarters. Community Dent Oral Epide-

mi011995;23:84-8.20. Pindborg JJ. Scree ning for oral cancer. In:Proro k PC, Miller AB, eds . Screen ing forcancer. Technical Report Series, 78 . Ge -neva: lntemational Union Against Can-cer, 1985.21. Bruck TL. Dental screening. In: Franker-b u rg W K , C am p B W , ed s . P ed ia t r i cscreening tests. Springfield, IL: CharlesC. Thomas, 1975:270-84.22. Piehla T, Retila I, Vaataja P. Early screen-in g for orthodontic treatment. Differ-ences in assessments mad e by a consult-ant orthodontist and three public healthden t ists . Community I k n t Oral Ep ide-miol 1992;20208-13.23. Gerstn er GE, Clark GT, Goulet J-P.Valid-ity of a brief questionnaire in screeningasymptomat ic sub jects f rom sub jectswith tension-type headachesor te mp or ema ndb u lar d iso rders . Commun ity Den tOral E pidem iol 1994;22:235-42.24. Association of State and Territorial Den-tal Directors, Maternal an d Child Hea lthBureau . Assess ing o ral heal th needs .ASTDD seven-s tep model . Co lumbus,OH: ASTDD, 1995.25. Heine CS, Jo ngA, Casamassimo PS,0sterbrock N, Call RL. Oral health statusand behav io rs of the elderly and otherad u l t s : r e su l t s of a heal th screen ing .Geriatr Dent 19833:217-21.26. Hu gh es JT, Rozier RG, Ramsey DL.Natu ral h is to ry of den tal d iseases inNorth Carolina 19761977. Durham, NC:Carolina Academ ic Press, 1982.27. Maine Departme nt of Human Services.Survey of the oral health of Maines hfth-grade public schoolchildren. Augusta,ME: Maine D epartment of 1 luman Serv-ices, 1985.28. South C arolma D epartm ent of Healthand Environmental Control. The SouthCaro l ina Den tal I leal th and Pehatr icBlood Pressure Survey 1982-1983. C elumbia, SC:South Carolina Depa rtmentof Heal th and Env ironmental Con tro l.29. Mau riello SM, Bader JD , Disney JA,Graves RC. Examiner ag reement be-tween hygenists and dentists for cariesprevalence examinations. J Public t lealthDent 1990;5032-7.30. Disney JA, A berna thy JK , Graves RC,Mauriello SM, Bohannan I IM, Zack DD.Comparative effectiveness of visual/tac-tile and simplified screening examina-tions in caries risk assessment. Com mu-nity Dent Oral Epidemiol1992;20:32&32.31. Bovonsantijid Y. Factors asso aa ted withthe reliability of oral examination byteachers in Phuket primary school orald w a s e w e l l l a n c e p ro g ra m ( d is se rt a-tion). Bangkok, Thailand: Mahidol Uni-versity, 1990.32 . Wagener DK, Nou ja h P, I Iorowitz AM.Trends in cluldhood use of dental carep ro d u c t s co n ta in in g f lu o n d e : U n i t edStates, 198S89. Advance d ata from Vitaland Hea lth Statistics; no 219.1 lyattsville,M D Nat ional Cen ter fo r Health S ta t istics,1992.33.Cleveland J, Holm K, Malvitz D, i Iines B.Recent dental exams am ong adults inWashington State-Behavioral Risk Fac-tor Surveillance System, 1993-1994 [ A bstract]. J Dent Res 1996;75(Speclss):231.3 4 . K l e i n b a u m D G , K u p p e r L L, M o r -

213g en s t e rn H. Epidemio log ic research .Pr incip les and quan t i ta t ive methods.New York : Van Nostrand Reinho ld ,1986.35 . Ro th man KJ. Mod ern ep idemio logy .Boston, MA: Little, Brown and Com-pany, 1986.36. Heloe LA. Comparison of dental healthdata ob tained from questionnaires, inter-view s and clinical examinations. Scand JDent Res 1972;80495-9.37. Wid strom E, Nilsson B. Dental healthand p erceived treatement need s of Finn-ish immigrants in Sweden. %and J SocMed 1984;12:129-36.38. Kononen M , Lipasti J, Murtomaa 11 .Comparison of dental informahon o btained from self-examination and c hi ca lexamination. Com mun ity Dent Oral Epi-demiol1986;14:258-60.39. Tervonen T, Knuuttila M . Awareness ofd en ta l d i so rd e r s an d d i screp an cy b ctween objective and subjective den-tal treatment needs. Community DentOral Epidemiol1988;16:3458.40.Lahti S, Tuutti H, Honkala E. C o m p an -son of numb ers of remaining teeth fromquestionnaires a nd clinical examination.Proc Fin Dent SOC 1989;85217-23.41. Palmqvist S, Sbderfeldt B, Arnbjerg D.Self-assessment of dental condtions: va-lidity of a questionnaire. CommunityDent Oral Epidemiol1991;19:249-51.42. Douglass CW, Berlin J, Tennstedt S. Thevalidity of self-reported oral health statu sin the e lder ly . J Publ ic Heal th Den t1991 51:220-2.43. Beal JF, Dickson S.Parental aw areness ofthe dental needs of 5-year-old children inthe West Midlands, England. Commu-nity Dent O ral Epidemiol 1974;2:914.

44. enters for Disease Control and Preven-t ion . Recommended in fect ion -con tro lpractices in den tistr y , 1993. MMWRMorb Mortal W kly Rep 1993;42(RR-8).45. Summ ers CJ, Gooch BF, Marianos DW,Malvitz DM , Bond WW . Practical infec-tion control in oral health surveys andscreenings. J Am Dent Assoc 1994;125:1213.17.46.Kaste LM, Gift HC, Bhat M, Sw ang o PA.Prevalence of incisor trauma in persons 6to 50 years of age: United States, 198%1991. J Dent Res 1996;75(Spec Iss):696705.47. Dean AG, Dean JA, Burton Afl, UlckerRC. Epi Info, Version 6. Atlanta: Ce ntersfor Dse ase Control an d Prevention, 1994.48. Gordis L. Epidemiology. Philadelphia:WB S u n d e r s , 1 99 6.49 . FleissJL.Statisticalmethodsforratesandproportions. New York: John Wiley &Sons, 1981.50. Landis JR, Koch GG. The measurementof observer agreement for categoricalda ta. Biometrics 1977;33:159-74.51. Alderman EJ, White SL, Johnson WT.Georga dental disease prevalence SUT-vey. Georgia Dent Assoc Action 1994:17-19 .5 2. C a m p BW. h c i p l e s of screening. In:Frankerburg WK, Camp BW, eds. Pedi-at r ic screen ing tests. Springfield, IL:Char les C. Thomas, 19755-8.53. Gupta PC, Mehta FS,Daftary DK, et al.Incidence rates of oral cancer and naturalhistory of oral precancerou s lesions in a


9/9

214 Journa l of Public Health Dentistry10-year follow-up study of Indian mllag-ers. Community Dent Oral Epiderniol1980;8:283-33.

54 . Bouquot JE , Gorlin RJ. Leukoplakia, li-chen planus, and other oral keratoses in23,616 white Americans over the age of35 years. Oral Surg 1986;61:373-81.55.Axd l T. Occurrence of leuk oplaha andsome other oral white lesions among20,333 adult Swemsh people. Commu-nity Dent Oral Epidemiol 1987;13:46-51.

oral cancer using basic health workers inan area of h g h oral cancer inadence inIndia. Cancer Detect Prev 1986;9218-25.57. Fenlon MR, McCartan BE. Validty of apatient self-completed health question-naire in a primary care dental practice.Community Dent Oral Epidemiol 1992;20:130-2.58. CYSullivan DM, Tinanoff N . Maxillaryanterior caries associated with increasedcaries risk in other primary teeth. J Dent

The use of rapid enzymatic assay in thefield for the detection of infections as=aa ted with adult periodontitis. J PublicHealth Dent 1993;53:2354.60. Brunette DM. Critical thinlung. Under-standing and evaluating dental research.Chicago, IL: Qluntessence, 1996.61. Shapiro MF, Lehrnan AF. he dagnosisof depression in different clinical set-tings. An analysis of thelitera ture on thedexamethasone suppression test. J Nerv56. Mihta FS, Cupti PC, Bhonsle RB, MurtiPR, Daftary DK, Pindborg JJ . Detection of Res 1993;72:1577-86. Ment Dis 1983;171:7i4-20.59. Bretz WA, Eklund SA , Radicchi R, et al.

ALAMANCE COUNTYHEALTH DEPARTMEN TNORTH CAROLINADentist position available with the Alamance CountyHealth Department, Burlington, NC. Duties include:acting as director for Dental Clinic; determining den-ta l needs; treatment plans; providing restorative, sur-gical, and preventive care for children enrolled in theDental Clinic; serving as consultant for public healthdentistry in Alamance County, working closely withschools, area dentists, and community agencies; beingresponsible for the opera tion and fiscal control of theDental Program, including supervision of 3-5 staffmembers.Salary commensurate with experience. Paid em-ployee medical/life/dental/liability insurance; de-pendent insurance available at group rates; paid sickand annual leave; 11-12 paid holidays; state retire-ment plan; 401K (employee contribution only).Requirements: graduation from an accredited schoolof dentistry with a license to practice dentistry in NC.Send completed county application to Becky Perkins,Alamance County Health Department, 319 N. Gra-ham-Hopedale Road, Burlington, N C 27217. Tel.:(910) 513-5517 for more information.

FUTURE AAPHDANNUAL MEETINGDATESOctobe r 21-23/1998. . . . .San Francisco, CAOctober 6-8,1999. . . . . . . . . . .Honolulu, HIOctober 25-27/2000 . . . . . . . . . . .Chicago, IL

COLUMBIA UNIVERSITYSYMPOSIUM TARGETS MED ICAIDMAN AGED CARE"Medicaid Managed Care and its Rela tionship to Den-tistry" will be the topic of the 13th Dunning MemorialSympsium held at Columbia University School ofDental and Oral Surgery on March 27,1998. Initiatedin 1981 by James Dunning, dean emeritus at HarvardSchool of Dental Medicine, the symposium is dedi-cated to the memory of William and Henry Sage Dun-ning, cofounders of the Columbia University School ofDental and Oral Surgery.Focusing on dentistry in the era of managed care, thesymposium will include opening remarks by the Hon-orable Barbara A. DeBuono, commissioner of health ofthe New York Department of Health, as well as pres-entations from both the private and public sectorsgiven by Dr. Robert Isman, dental program consultantwith the California Department of Health Servicesandproject director of the Children's Dental Health Initia-tive of the Dental Health Foundation, and Dr. Alex B.White, current president of the AAPHD and seniorinvestigator and associate program director of healthservices and social and economic studies at the KaiserPermanente Center for Health Research. The after-noon will involve presentations by three New YorkState Provider Grant Recipients: DentNY IPA, Inc.,Neighborhood Health Providers, and Syracuse Com-munity Health Center, Inc. These will reflect analysesof experiences with populations being served by themanaged care plans. Professor Emeritus Irwin D.Mandel will then lead a panel discussion that willinclude the program participants and Dr. JamesSpencer, a member of the ADA Council on DentalBenefit Programs.The symposium is sponsored by the School of Dentaland Oral Surgery of Columbia University, the Dun-ning Memorial Fund, and the New York Council ofDental Deans. Continuing education credits will beawarded. For further information, please contact:Melissa Welsh at Columbia University: (212)305-6881.

Documents

Validity Oral