Comparative evaluation and analysis of online geocoding services

Comparative evaluation and analysis of online geocoding services

Duangduen Roongpiboonsopit and Hassan A. Karimi*

Geoinformatics Laboratory, School of Information Sciences, University ofPittsburgh, Pittsburgh, PA, USA

(Received 18 October 2008; final version received 23 August 2009)

Geocoding is an uncertain process that associates an address or a place name withgeographic coordinates. Traditionally, geocoding is performed locally on a stand-alonecomputer with the geocoding tools usually bundled in GIS software packages. The use ofsuch tools requires skillful operators who know about the issues of geocoding, that is,reference databases and complicated geocoding interpolation techniques. These days, withthe advancement in the Internet andWeb services technologies, online geocoding providesits functionality to the Internet users with ease; thus, they are often unaware of such issues.With an increasing number of online geocoding services, which differ in their referencedatabases, the geocoding algorithms, and the strategy for dealing with inputs and outputs, itis crucial for the service requestors to realize the quality of the geocoded results of eachservice before choosing one for their applications. This is primarily because any errorsassociated with the geocoded addresses will be propagated to subsequent decisions,activities, modeling, and analysis. This article examines the quality of five online geocod-ing services: Geocoder.us, Google, MapPoint, MapQuest, and Yahoo!. The quality of eachgeocoding service is evaluated with three metrics: match rate, positional accuracy, andsimilarity. A set of addresses from the US Environmental Protection Agency (EPA)database were used as a baseline. The results were statistically analyzed with respect todifferent location characteristics. The outcome of this study reveals the differences amongthe online geocoding services on the quality of their geocoding results and it can be used asa general guideline for selecting a suitable service that matches an application’s needs.

Keywords: location-based services; Internet GIS; quality issues; uncertainty; Web GIS;geocoding

1. Introduction

In the information age, spatial information is considered as one of the most essential value-added pieces of information. The simplest way to obtain spatial data is to interpret addressessince addresses are the fundamental means people use to describe locations. The process toconvert addresses into geographic locations is known as geocoding, which is a basicoperation in geographic information systems (GISs) (Antenucci et al. 1991, Huxhold 1991,Longley et al. 2005). In early GISs, the use of geocoding was limited to groups ofprofessionals who had a good understanding of the underlying processes and associateduncertainties. However, with the advent of the Internet and the advancement inWeb servicestechnologies, the complexity of geocoding has become transparent to end users. Nowadays,converting an address into a set of coordinates on the fly can be accomplished by submitting arequest to an online geocoding service with little knowledge, little effort, and little or no cost.

International Journal of Geographical Information ScienceVol. 24, No. 7, July 2010, 1081–1100

*Corresponding author. Email: [email protected]

ISSN 1365-8816 print/ISSN 1362-3087 online# 2010 Taylor & FrancisDOI: 10.1080/13658810903289478http://www.informaworld.com

mailto:[email protected]

http://www.informaworld.com

Currently, several online geocoding services are available to users, in particular to devel-opers, who want to incorporate spatial data in their Web-based applications. Such applicationsrange from simply visualizing address locations on a map to complex spatial analysis. BesidestypicalWeb-based applications, location-based service (LBS) applications, especially on mobilephones, have been emerging exponentially. LBS applications typically rely on a geocodingservice to find points of interest, to search for nearby businesses, and to compute routes amongother functions. However, even though geocoding services have improved in terms of accessi-bility and availability, there are still uncertainties associated with the results they produce(Karimi et al. 2004, Goldberg et al. 2007, Zandbergen 2008). As geocoded data are a funda-mental input into many applications, geocoding uncertainties will be propagated to subsequentprocesses and will cause inaccurate analysis results and unreliable decisions by users.

Many studies, mostly in public health and epidemiological research, have compared thequality of geocoded results assigned by several geocoders. These studies vary from havingfull control over the geocoding algorithm and the reference database to having a completelyblind process of geocoding and uncontrolled data sets. The empirical comparisons wereusually conducted among different commercial geocoding vendors (Krieger 2001, Whitselet al. 2004, 2006), among in-house software packages such as ArcView and Automatch(Yang et al. 2004), or between commercial geocoding vendors and commercial softwarepackages (Ward et al. 2005, Lovasi et al. 2007, Strickland et al. 2007).

Common metrics for evaluating the quality of geocoding results are completeness (ormatch rate), positional accuracy, and repeatability (Zandbergen 2008). In general, thegeocoding quality depends on several factors such as geographic areas of addresses, qualityof the reference databases, match scores, and geocoding algorithms. Several studies haveshown that addresses in rural areas are far less accurate and have a much lower match ratethan those in urban areas (Bonner et al. 2003, Cayo and Talbot 2003, Ward et al. 2005), andresidential addresses have relatively higher match rates than commercial addresses(Zandbergen 2008). Meanwhile, Karimi et al. (2004) have shown that there is no significantdifference among the three geocoding algorithms (LocMatch, ArcView, and TANA) ongeocoding accuracy. Zandbergen and Green (2007) found that the positional errors in streetgeocoding often occur on one side of the street.

The study presented in this article is different from the previous studies in several ways.First, we focus mainly on evaluating the results of the following online geocoding services:Geocoder.us, Google, Microsoft, MapQuest, and Yahoo!. The use of these services iswidespread and freely available to all users. Although our approach in comparing servicesis somewhat similar to Krieger (2001) and Whitsel et al. (2004, 2006), our results arebeneficial to a wider range of users. Since the previous studies do not identify whichgeocoding services they are evaluating, users are not able to select the best one for theirneeds based on those studies. Second, this study employs the two common metrics of matchrate and positional accuracy and introduces a new metric, called similarity, to find common-ality of the results among the services. Repeatability, a measure of sensitivity to variations inthe reference database and the underlying algorithms, was not considered in our studymainly because service providers do not update their algorithms and reference databasesfrequently. Third, this study provides up-to-date results from the current geocoding technol-ogies, including the underlying algorithms and reference databases. Fourth, this studyhighlights the different requirements and behaviors of online geocoding services, particu-larly request and response messages, which are important issues for developers or research-ers who want to use several services in combination.

In summary, the contributions of this article are a comprehensive comparison of widelyused online geocoding services, with the objective of better understanding the quality of

1082 D. Roongpiboonsopit and H.A. Karimi

online geocoding services, and a set of guidelines for selecting a suitable online geocodingservice for the underlying application. The rest of the article is organized as follows. Thenext section gives a brief background on conventional and online geocoding. Section 3provides an overview of the five geocoding services and the baseline used in this study.Section 4 describes the method for performing analysis, and the evaluation results arepresented in Section 5. Section 6 discusses the analysis of results and the article ends withconclusions in Section 7.

2. Background

2.1. Conventional geocoding

Geocoding is the process of assigning the geographic code (e.g., coordinates) to a givenplace name by comparing its description to the descriptions of location-specific elements inthe reference database (Karimi et al. 2004, Goldberg et al. 2007, Zandbergen 2008). Thesoftware performing the geocoding process is called a geocoder, which can be divided intofour components: input, output, processing algorithm, and reference database (Goldberget al. 2007). The input is a description of a feature or a place that has spatial information theuser wishes to convert to geographic coordinates and the output is the geographical referencecode. The core logical component is the processing algorithm. Generally, the steps of thegeocoding process include translating an input entry (Churches et al. 2002, Gu et al. 2003,Yang et al. 2004), searching for matched attributes in the reference database (Drummond1995, Christen et al. 2004, Yang et al. 2004, Goldberg et al. 2007), estimating possiblelocations, and delivering the best candidate(s) as the output. The reference database containsgeographical reference elements used by the processing algorithm to derive the output.Several address models exist for building the reference database, such as the street networkdata model, the geographic unit model or the parcel boundaries data model (e.g., postalcodes, counties, cities, census enumeration areas), and the address point data model(Zandbergen 2008).

2.2. Online geocoding

An online geocoding service is a network-accessible component, sometimes a module of aGIS, which automatically performs the geocoding process. It is usually available on theInternet by utilizing a Web service interface. The data entry, such as a place name, streetaddress, or zip code, is passed over the Internet using a communication protocol to thegeocoding service. The overall process usually takes less than a few seconds per data entry.The geocoding service converts an entry into coordinates and then delivers the result, whichincludes the coordinates, the address used in geocoding, and the level of accuracy back to theuser over the Internet. The underlying algorithm and reference database used by the serviceare transparent to the user. The term ‘online geocoding’ is used in this article, although otherterms, such as real-time geocoding, address lookup, and address matching service, have beenused interchangeably by the GIS community. In our definition, we exclude the services thathave human intervention during the geocoding process or do not deliver immediate geo-coded results to the users; for instance, the four geocoding services offered by commercialgeocoding firms in Krieger (2001).

Using online geocoding is different from using conventional geocoding tools that comewith GIS software packages, such as ArcView and Automatch, in several ways. First, onlinegeocoding service is easy to use. As illustrated in Figure 1, the user of online geocoding does

International Journal of Geographical Information Science 1083

not need to know how geocoding works or how to acquire and maintain the referencedatabase for the target area. All the user needs to do is input addresses, as required by theservice, and interpret the results. Second, the reference database in any online geocodingservice is stored, maintained, and updated by the service provider. Unlike conventionalgeocoding that puts the burden of providing reference databases on the user’s shoulder,online geocoding does not require that the user be concerned with reference databases.However, this could be a disadvantage as there are no provisions in online geocoding foradding, correcting, and updating elements in the reference databases. Third, service provi-ders are not only responsible for maintaining reference databases, but also for improving thegeocoding software. Typically, most service providers strive to provide high-quality geo-coding results by applying advanced algorithms or matching methods in order to becompetitive with other geocoding services. Service providers predefine all parameters andmethods of the geocoding process; as a result, they do not allow the users to customize suchsetting as match scores and relaxation rules. Fourth, certain applications heavily rely on timeperformance, which in part is affected by the Internet infrastructure. Latency in the geocodedresult could delay the succeeding analysis or processes performed in Web applications orlocation-based applications. Table 1 summarizes the advantages and disadvantages of usingonline geocoding services from the user perspective.

3. Overview

3.1. Testing services

In this study, we selected five well-known and publicly accessible services, ranked amongthe top providers of geocoding services. Table 2 summarizes the characteristics of each

Table 1. Summary of the advantages and disadvantages of using online geocoding services.

Advantages Disadvantages

1. Easy to use 1. No control over the reference database2. Immediate coordinate results 2. No control over the parameter of geocoding

process (e.g., match score, relaxation rules)3. The user does not need to acquire, maintain, andupdate the reference database

3. Unknown quality of geocoded results

4. No software or tool is required on the user side 4. Relying on the Internet infrastructure

Figure 1. The high-level process of (a) conventional geocoding and (b) online geocoding.


service. Of these five geocoding services, Google1 and Microsoft2 offer a unique feature,called rooftop geocoding, for improving the accuracy of traditional geocoding results for USaddresses. The concept of rooftop geocoding is not new. Rooftop geocoding is an interpola-tion technique that employs the address point data acquired from imagery as the referencedata. Each address point represents the roof of a particular address. This is in contrast tostreet geocoding, a widely used geocoding technique, where information on road segmentsis employed to geocode addresses. All five services provide both noncommercialand commercial versions. However, these services do not report the differences in qualityof the geocoding results produced by the different versions. In this article, we evaluated onlythe noncommercial services and used the information available on their websites, at the timeof writing.

3.2. Baseline data

We define geocoding baseline as any existing geocoded address along with its measuredquality. Originally, we intended to evaluate the performances of online geocoding servicesfor the global geographic extent. However, due to limited access to accurate and reliableglobally covered baseline data, we narrowed the scope of this study to the United States. Wesearched several publicly available electronic data sources. Among them, we chose theEnvironmental Protection Agency (EPA) Air Quality Systemmonitors’3 database because ofits consistency between the sites’ addresses and their corresponding coordinates. However,using the EPA database has a shortcoming. The coordinates reported by the EPA are thelocations of the instruments. This introduces an uncertainty about the actual location of eachaddress. While the parcel’s centroid, the mailbox’s location, and the middle of street frontageare alternatives, they are unavailable for large areas or may not be available in one database,

Table 2. Summary of selected online geocoding services.

Service name CompanyLimitation of noncommercial

service Service interface Coverage

Geocoder.usa LocativeTechnologies

No limitation has beenexplicitly regulated in thelicense agreement

SOAP; XML-RPC;REST-CSV, RDF

USA

Google Maps’geocodingserviceb

Google 15,000 requests per IP per day JavaScript API;REST-CSV,JSON, or XML

Global

MapPoint WebServices’FindAddressc

Microsoft 1000 transactions per day SOAP Global

MapQuest’sgeocodingserviced

AmericaOnline, Inc.(AOL)

No limitation has beenexplicitly regulated in thelicense agreement

JavaScript API;REST-XML

Global

Yahoo! Maps’services –geocoding APIe

Yahoo! Inc. 5000 queries per IP per day REST-XML Global

ahttp://geocoder.us/bhttp://code.google.com/apis/maps/documentation/services.html#Geocodingchttp://www.microsoft.com/mappoint/products/webservice/default.mspxdhttp://developer.mapquest.com/homeehttp://developer.yahoo.com/maps/rest/V1/geocode.html


ahttp://geocoder.us

bhttp://code.google.com/apis/maps/documentation/services.html#Geocoding

http://www.microsoft.com/mappoint/products/webservice/default.mspx

http://developer.mapquest.com/home

http://developer.yahoo.com/maps/rest/V1/geocode.html

and collecting them is time consuming and expensive. In utilizing the EPA data, we used thecoordinates of the sensors to represent the location for that address.

The EPA database contains a total of 6148 sites, of which we selected only 552 sites thatmet the following criteria:

(1) The site must contain a full street address (no PO Boxes, building names, andintersections).

(2) The coordinates of the site’s location were collected by any technique (e.g., GPS,interpolation) except address matching.

(3) The horizontal accuracy of the site’s coordinates is greater than 0 and less than orequal to 5 m.

(4) The datum associated with the coordinate must be known (e.g., WGS84, NAD83, orNAD27).

Figure 2 shows the coordinates, reported by EPA, of the 552 selected addresses. Alltypos were corrected and the outdated city names were updated. Note that due to the missingzip code information in some of the EPA addresses, we decided not to parse zip code to thegeocoding services so that biases from different address formats could be avoided.Addresses were submitted using the following format: ‘house number, street name, city, state’.

The EPA data also provide information about location setting and land use related to eachsite. Three types of location setting are associatedwith the selected sites: urban (185), suburban(229), and rural (137). One site was unknown. There are seven types of land use but only fourof them have adequate sizes for performing analysis: residential (233), commercial (179),agricultural (74), and industrial (58). This additional information will allow us to shed light onthe impact of the two groups of different address types on the geocoding results.

4. Method

A total of 552 EPA addresses were submitted to the five online geocoding services:Geocoder.us, Google, MapPoint, MapQuest, and Yahoo!. The returned results were storedin a PostgreSQL database for analysis. We evaluated and compared the performances of thefive geocoding services on match rate, positional accuracy, and similarity. Match rate is the

Figure 2. Locations of the 552 testing addresses.


number of geocoded addresses divided by the total number of submitted addresses.Positional accuracy is measured by the error distance that the geocoded point deviatedfrom the baseline. The shorter the error distance, the higher the positional accuracy of thegeocoded points. Moreover, we evaluated similarity among services on the geocoded points.Similarity is measured by pairwise distances between pairs of the geocoded points. As aresult, the shorter the pairwise distances, the more likely the two geocoding services produceequivalent geocoded coordinates. The two types of distances are the ellipsoidal distances,calculated between pairs of latitude/longitude coordinates in WGS84 datum. The coordinatesnot in WGS84 are transformed before performing the distance operation. Some baselinepoints collected by EPA are in NAD83 and NAD27 and the Geocoder.us’s geocoded pointsare in NAD83. These were transformed to WGS84 to avoid subtle differences among thethree reference datums.

5. Results and analysis

5.1. Match rate

The overall match rates returned by Geocoder.us, Google, MapPoint, Map Quest, andYahoo! were 73.19%, 96.56%, 92.39%, 100%, and 100%, respectively. For analysis pur-poses, we classify the quality of geocoded results into three levels by taking advantage of thegeocoding reports attached to the result. The three levels are ‘good’, ‘ambiguous’, and‘unmatched’. Because there is no consensus on how to report the quality of geocoded results,we sorted their codes or messages into the groups as presented in Table 3. ‘Good’ matchesgenerally mean geocoded points with high confidence at the address level and no warningmessages. ‘Ambiguous’ matches mean geocoded points with lower confidence at street, city,

Table 3. Codes of the geocoded results for each geocoding service.

Type ofmatches Geocoder.us Googlea MapPointb MapQuestc Yahoo!d

Good Housenumbermatchesthe parsedaddress

Accuracy = 8(addresslevel)

Match code = 0(good) andmatchedcode = 0(interpolationaddress level),7 (rooftop)

Code L1AAA (findone match)

Match at‘address’levelwithout awarningmessage

Ambiguous Housenumberdoes notmatch theparsedaddress

Accuracycode = 6(streetlevel)

Match code = 2(Imprecise)and matchedcode = 1 (streetlevel)

Code L1BAA (findwith street namecorrection) codeB3AAA, B3BAA(B3 nearest blocklevel), A5XAX,A5XCX (citylevel)

Match at‘address’level with awarningmessage,‘street’level, and‘zip’ level

Unmatched No geocodedresult

No geocodedresult

No geocodedresult

No geocoded result No geocodedresult

ahttp://code.google.com/apis/maps/documentation/reference.html#GGeoAddressAccuracybhttp://msdn.microsoft.com/en-us/library/cc546335.aspxchttp://developer.mapquest.com/documentation/content/ApiDocumentation/Protocols/MQAE_XML_Reference.pdfdhttp://developer.yahoo.com/maps/rest/V1/geocode.html


http://code.google.com/apis/maps/documentation/reference.html#GGeoAddressAccuracy

bhttp://msdn.microsoft.com/en-us/library/cc546335.aspx

chttp://developer.mapquest.com/documentation/content/ApiDocumentation/Protocols/MQAE_XML_Reference.pdf

dhttp://developer.yahoo.com/maps/rest/V1/geocode.html

zip, state, or address level with warning messages. Table 4 reports match rates classified bythe quality levels and address types.

The quality of the matched results returned by each service was compared using theFriedman test, a two-way analysis on ranks, by assigning 0, 1, and 2 to good, ambiguous, andunmatched matches, respectively. If the null hypothesis is true, the mean rank of each serviceis expected to be (5 + 1)/2 = 3. Figure 3a shows the means of the ranked measures computedby the test for all testing addresses. The test revealed a significant difference (Q = 270.61,p , 0.05) based on ranks. A post hoc analysis using Wilcoxon sign tests (p , 0.005)revealed that overall Google, MapPoint, and Yahoo! provided statistically better qualitymatches than MapQuest and Geocoder.us, whereas Geocoder.us provided the least number ofmatches.

Analyzing the impact of location setting on the quality of matches, three Friedman testswere performed using the adjusted confidence level = 0.05/3 = 0.017 to reduce Type I error.The analysis revealed that all three tests found a significant effect among different geocodingservices on the quality of matches (Q = 104.7, 97.80, 85.43 for urban, suburban, and rural,respectively; p , 0.017). Figure 3b shows the plot of mean ranks calculated by the threetests. Post hoc tests showed that Geocoder.us was significantly worse (p , 0.0017) than theother services for all types of location setting. In addition, we found that Google wassignificantly better than MapQuest for urban areas and Google, MapPoint, and Yahoo!were significantly better than MapQuest for suburban areas.

Furthermore, we analyzed the impact of four types of land use. Four Friedman tests wereconducted at a = 0.05/4 = 0.0125. Figure 3c shows the means of the rank measures computedby the four tests. The analysis revealed that all four tests were significant (Q = 89.37, 83.88,

Table 4. Match rates of geocoded results.

Match rate (%)

Count Match level Geocoder.us Google MapPoint MapQuest Yahoo!

All data 552 Good 60.69 88.04 86.05 69.20 78.44Ambiguous 12.50 8.52 6.34 30.80 21.56Unmatched 26.81 3.44 7.61 – –

Locationsetting

Urban 185 Good 67.57 95.14 94.59 81.62 85.41Ambiguous 8.11 2.70 2.16 18.38 14.59Unmatched 24.32 2.16 3.24 – –

Suburban 229 Good 65.50 92.14 86.90 69.43 80.35Ambiguous 14.41 6.99 5.68 30.57 19.65Unmatched 20.09 0.87 7.42 – –

Rural 137 Good 43.07 71.53 72.99 51.82 65.69Ambiguous 15.33 18.98 13.14 48.18 34.31Unmatched 41.61 9.49 13.87 – –

Land use Res. 222 Good 68.47 93.24 90.99 75.23 80.18Ambiguous 8.56 4.05 5.41 24.77 19.82Unmatched 22.97 2.70 3.60 – –

Comm. 178 Good 64.04 91.57 87.64 70.79 81.46Ambiguous 13.48 4.49 2.81 29.21 18.54Unmatched 22.47 3.93 9.55 – –

Agri. 74 Good 36.49 72.97 75.68 48.65 68.92Ambiguous 20.27 20.27 12.16 51.35 31.08Unmatched 43.24 6.76 12.16 – –

Ind. 57 Good 54.39 84.21 82.46 70.18 75.44Ambiguous 14.04 14.04 14.04 29.82 24.56Unmatched 31.58 1.75 3.51 – –


62.16, 38.97 for residential, commercial, agriculture, industrial; p , 0.0125). A post hocanalysis showed that Geocoder.us was significantly worse (p , 0.00125) than the otherservices for the four types of land use. Additionally, we found some differences among theservices for residential, commercial, and agricultural addresses but the results were notconsistent; thus, we do not present those results here. However, with industry-type addresses,Geocoder.us provided the worst quality of matches.

Figure 3. The means of the ranked measures computed by the Friedman test on the match ratesclassified by (a) all addresses, (b) the addresses with location setting, and (c) the addresses with land use.


5.2. Positional accuracy

The error distances of all geocoded addresses produced by each service are illustrated inFigure 4. We eliminated outliers of each service whose distances are beyond�3 times of itsinterval for variance. The descriptive statistics for all data and for data without outliers arepresented in Table 5. Table 6 summarizes the quality of positional accuracy in terms of errordistances, sorted by service names and address types.

Because the data are positively skewed and have unequal variances, we transformed thedata by using a logarithm transformation. We performed the analysis on 386 addresses thatwere geocoded by all five services to avoid biases frommissing data. Figure 5 shows the boxplot of the log-transformed error distances for these 386 geocoded addresses. Although thesetransformed data satisfied homoscedasticity, the normality assumption was violated(p , 0.05 in Kolmogorov–Smirnov and Shapiro–Wilk test). As a result, the Friedman testbased on ranks was chosen.

The test revealed a significant difference on ranks (Q = 79.86, p , 0.05). Figure 6ashows the medians of the error distances for each service (left) and the means of the rankedmeasures computed by the test (right). Post hoc tests were conducted at the adjusted level,p = 0.005, for each pairwise test. After combining the 10-test results, the services can beclassified into two groups. Yahoo!, MapPoint, and Google produce more accurate geocodedcoordinates than Geocoder.us and MapQuest.

As the difference among geocoding services was significant, we further examined theimpact of different address types, that is, location setting and land use, on accuracy.

A total of 138 urban, 176 suburban, and 71 rural addresses were geocoded by allservices. Three Friedman tests were conducted at the adjusted confidence level, a = 0.05/3 = 0.0167, to avoid Type I error, since the same set of results has been used in the overallanalysis previously. The test for the rural addresses was not significant (Q = 4.18,p . 0.0167). In contrast, the tests for the urban and suburban addresses were significant(Q = 30.65, 59.27, respectively, p , 0.0167). Figure 6b shows the median plot of errordistances for each service (left) and the mean rank plot computed by the three tests (right).Ten pairwise comparisons following the omnibus tests of the urban and suburban addresseswere conducted at a = 0.0167/10 = 0.00167. For urban addresses, the geocoded pointsreturned by Yahoo!, Google, and MapPoint were significantly more accurate than that ofMapQuest, and only Yahoo! provided significantly higher accuracy than Geocoder.us.Similar to the urban case, the geocoded points of the suburban addresses returned byYahoo!, Google, and MapPoint were significantly more accurate than those of MapQuest.

Figure 4. The distribution of error distances computed between each geocoded point and its corre-sponding baseline for each service.


Table5.

The

descriptivestatistics

ofpo

sition

alaccuracy

foralldataanddatawitho

utou

tliers.

Alldata

Datawitho

utou

tliers

Service

name

Cou

ntMin

(m)

Max

(m)

Median(m

)Mean(m

)STD(m

)Cou

ntMin

(m)

Max

(m)

Median(m

)Mean(m

)STD(m

)

Geocoder.us

404

0.2

26,387

.120

8.3

887.6

2,62

9.6

400

6.8

18,332

.720

7.1

723.8

1,79

7.4

Goo

gle

533

5.5

1,54

0,27

8.3

182.2

9,20

4.3

91,328

.952

45.5

44,537

.417

8.5

1,00

0.1

3,60

3.6

Map

Point

510

8.9

347,20

7.2

172.5

2,42

2.0

22,002

.850

38.9

12,966

.017

1.0

436.4

1,07

0.8

Map

Quest

552

0.3

347,20

0.3

242.1

4,01

0.5

24,249

.154

46.8

59,926

.723

8.4

1,47

4.0

4,64

6.7

Yahoo

!55

22.1

2,09

8,07

2.8

173.4

6,55

1.8

91,792

.054

52.0

29,534

.217

2.6

748.4

2,40

9.2


Additionally, the test showed that Yahoo! provided significantly higher accuracy thanMapPoint and Geocoder.us, and MapPoint was significantly better than Geocoder.us.

A total of 168 residential, 134 commercial, 36 agricultural, and 38 industrial addresseswere investigated to explore the impact of different land use types on positional accuracyproduced by all the geocoded services. In each of the four tests, the significance level wasadjusted at a = 0.05/4 = 0.0125. Figure 6c shows the medians of error distances for the fiveservices (left) and the means of the ranked measures calculated by the four tests (right). Thetests for agricultural and industrial addresses were not significant: Q = 2.98 and 5.32,respectively (p . 0.0125). On the other hand, the tests for the residential and commercialaddresses were significant, Q = 70.92 and 29.77, respectively (p , 0.0125). Post hoc tests(a = 0.00125) revealed that Yahoo!, Google, and MapPoint provided significantly moreaccurate results than MapQuest on the geocoded points of the residential addresses. Inaddition, Yahoo! returned significantly more accurate results than MapPoint andGeocoder.us, and MapPoint produced significantly higher accuracy than Geocoder.us. Forthe commercial addresses, Yahoo!, Google, and MapPoint provided significantly betteraccuracy than MapQuest and also Yahoo! and MapPoint produced significantly higheraccuracy than Geocoder.us.

It is also interesting to examine positional accuracy when the matching was identified asgood or ambiguous. According to Table 3, the error distances were sorted into two groups:good and ambiguous. To make the analysis consistent with the results of the match qualityanalysis, all data are included (no outlier elimination). Table 7 presents the descriptivestatistics of positional accuracy for both good and ambiguous matches.

Table 6. The quartiles of positional accuracy for each geocoding service.

Positional accuracy (m)

Count Quartiles Geocoder.us Google MapPoint MapQuest Yahoo!

Data withoutoutliers

386 1st 83.7 69.4 67.9 90.0 64.22nd 194.3 162.6 157.5 199.1 160.23rd 485.2 404.5 316.6 468.8 316.9

Location setting Urban 138 1st 68.5 55.0 55.4 75.4 47.12nd 159.3 129.3 134.0 165.8 142.03rd 306.0 269.2 243.9 313.6 269.0

Suburban 176 1st 90.1 72.0 68.9 90.2 64.92nd 191.8 157.3 143.5 185.7 148.13rd 495.5 329.9 274.9 460.5 260.7

Rural 71 1st 146.8 142.4 116.4 160.8 109.02nd 297.0 355.7 311.3 302.2 299.43rd 718.1 1,136.0 612.1 709.2 693.1

Land use Res. 167 1st 87.3 69.9 70.3 97.4 65.72nd 167.8 158.8 149.6 186.0 153.73rd 431.4 292.1 281.0 432.0 272.8

Comm. 134 1st 69.7 56.9 57.6 74.5 51.02nd 168.2 119.6 111.7 163.4 107.93rd 380.8 276.4 243.0 375.1 246.4

Agri. 35 1st 164.9 153.3 125.9 160.8 161.42nd 373.5 418.7 312.0 302.2 311.73rd 603.2 1,187.5 555.8 478.6 565.2

Ind. 38 1st 120.3 126.6 113.1 91.4 87.92nd 324.7 197.5 219.9 259.5 238.93rd 1,361.0 1,153.8 1,226.4 860.8 1,147.1


5.2.1. Good match

‘Good match’ error distances are positively skewed and have different variances, similar tothe overall results for positional accuracy. The logarithm function, log(x + 1) where x is theerror distance, was applied because the error distance can be less than 1 m. The transformeddata for each service, showed in Figure 7a, have about the same variance and a similar shapeof distributions, but they are not normally distributed. Thus, we chose the Kruskal–Wallistest, a one-way nonparametric test by ranks, to test equivalence of the population medians onpositional accuracy among geocoding services when the results were identified as ‘goodmatches’. The test was not significant (H = 5.841, p . 0.05). As a result, no major differencecan be found on positional accuracy of the geocoded coordinates identified as ‘goodmatches’ by the five testing services.

5.2.2. Ambiguous match

‘Ambiguous match’ data are also positively skewed and have different variances. Althoughthe log-transform function reduced the differences in their variances, the differencesremained rather high, as shown in Figure 7b. The Levene’s test, a test for homogeneity ofvariances, confirmed a significant difference with p = 0.009 (,0.05). Due to the violation ofhomoscedasticity, we conducted neither a parametric nor a nonparametric test to avoidunreliable results.

Figure 5. The log-transformed error distances of Geocoder.us (Gus), Google (G), MapPoint (MP),MapQuest (MQ), and Yahoo! (Y) for 386 addresses.


Figure 6. The left plot shows the median plot of error distances and the right plot shows the means ofthe ranked measures computed by the Friedman test on positional accuracy for (a) 386 addresses, (b)the addresses with location setting, and (c) the addresses with land use.

Table 7. The quartiles of positional accuracy for good and ambiguous matches.

Positional accuracy (m)

Match quality Quartiles Geocoder.us Google MapPoint MapQuest Yahoo!

Good Count 335 486 475 382 4331st 76.0 69.6 67.9 82.8 69.32nd 164.9 165.5 155.0 173.7 158.63rd 359.1 347.9 307.8 383.4 311.1

Ambiguous Count 69 47 35 170 1191st 317.1 417.5 345.1 230.2 112.92nd 980.3 3,416.4 842.5 901.5 262.23rd 3,477.0 9,227.2 2,058.1 3,707.4 1,478.5


5.3. Similarity comparison

Wemeasured the similarity among the five services on pairwise distances. This analysis usedonly the geocoded points of the 386 addresses that were returned by all services. We foundthat the data were severely skewed to the right and had unequal variances. We tried severalfunctions to transform the data but all transformations failed both normality and homosce-dasticity tests (p , 0.05). As a result, we could not apply the Friedman test for comparingsimilarity among services. Instead, we report the quartiles of similarities in Table 8 andpresent the statistical plot of the log-transformed data in Figure 8.

6. Discussion

For the overall match rates, Yahoo! and MapQuest returned geocoded results 100% of thetime, higher than Google, MapPoint, and Geocoder.us. This difference occurs due todifferent strategies used in the matching relaxation process. If the street name cannot bematched, Yahoo! and MapQuest choose to return coordinates from an upper level of theaddress hierarchy, in particular at the city level for MapQuest and at the zip code level forYahoo!, whereas Google and MapPoint choose to return nothing above the street level inorder to maintain high accuracy. Google has slightly higher match rates than MapPointpartially because Google does not limit the search for the given street name within the givencity. If Google fails to find a street-level match in the given city, it expands its search toinclude additional cities. Geocoder.us, on the other hand, when no address- or street-levelmatch is found, does not continue to attempt to find a match. Another possible cause for thelower match rates by Geocoder.us is the TIGER data files, used as its reference database,which could be incomplete and/or out of date for the testing areas.

After examining the quality of match rates produced by the five services, the percentagesof ‘good match’ rates from Google and MapPoint are higher than Yahoo!, MapQuest, and

Figure 7. The log-transformed error distances for geocoded results with good and ambiguousmatches producted by Geocoder.us (Gus), Google (G), MapPoint (MP), MapQuest (MQ), andYahoo! (Y).


Table8.

The

quartilesof

similarityam

ongeach

pairof

geocod

ingservices.

Cou

ntQuartiles

Sim

ilarity(m

)

Gus-G

Gus-M

PGus-M

QGus-Y

G-M

PG-M

QG-Y

MP-M

QMP-Y

MQ-Y

Datawitho

utou

tliers

386

1st

31.8

25.1

0.0

27.0

2.5

30.0

22.1

24.3

16.1

26.1

2nd

95.6

72.4

0.1

60.0

26.4

97.0

63.8

74.8

21.6

63.7

3rd

282.5

252.9

64.8

265.4

149.0

284.5

203.2

251.4

52.7

243.2

Locationsetting

Urban

138

1st

20.6

18.9

0.0

19.4

2.2

19.6

18.0

18.9

15.7

19.6

2nd

57.8

54.7

0.1

34.0

8.3

56.9

36.0

56.1

18.4

34.9

3rd

142.9

112.5

14.4

105.9

69.0

155.0

109.3

123.3

31.8

116.4

Sub

urban

176

1st

36.7

30.2

0.0

27.7

1.4

31.1

24.5

29.1

16.5

27.4

2nd

109.8

75.4

0.1

68.8

21.0

114.2

69.7

83.8

24.9

64.1

3rd

315.6

260.6

88.9

295.1

131.4

293.0

195.3

264.3

75.1

238.8

Rural

711st

68.0

36.4

0.0

53.0

21.3

74.4

54.6

48.4

16.8

59.4

2nd

183.9

200.4

1.3

173.2

161.5

149.5

209.9

183.9

31.2

194.8

3rd

834.3

592.6

221.4

468.0

1,13

8.8

1,09

6.9

1,06

7.0

526.8

102.9

468.0

Landuse

Res.

167

1st

29.8

23.0

0.0

24.0

2.4

29.8

23.0

23.4

15.8

24.4

2nd

74.3

64.3

0.1

42.2

21.4

94.1

59.0

72.7

21.1

51.4

3rd

187.5

172.0

38.5

132.0

138.5

244.2

175.8

209.8

47.3

188.5

Com

m.

134

1st

27.5

20.6

0.1

26.0

1.0

23.1

19.6

19.7

16.2

24.6

2nd

76.6

61.5

0.1

53.2

10.5

63.5

44.9

57.8

20.0

51.7

3rd

232.0

203.8

49.0

285.0

73.5

209.9

129.2

218.7

43.4

234.8

Agri.

351st

101.4

87.3

0.0

93.6

38.8

92.5

77.0

87.3

18.9

93.1

2nd

309.0

287.3

11.9

262.1

170.9

257.1

350.7

227.5

31.7

226.0

3rd

1,20

2.6

516.4

221.4

872.7

1,36

9.4

1,21

2.0

1,45

6.5

409.8

140.3

440.3

Ind.

381st

35.7

34.4

0.1

23.9

0.0

30.8

19.6

26.2

16.7

22.4

2nd

113.6

102.2

0.3

70.8

13.5

97.5

84.6

91.1

38.7

52.3

3rd

763.9

760.5

90.7

597.1

135.8

408.5

285.3

419.6

190.2

277.8


Geocoder.us. This is, in part, because Google and MapPoint provide a coarser level of detail(only accuracy level for Google and matched code for MapPoint) tagged to the geocodedresult. Neither provides warning messages or codes to identify the changes made onsubmitted addresses, such as typos, corrections of outdated city names, and prefix or suffixof street names. For instance, Google geocoded ‘321 Walker Lane, Huntsville, AL’ as ‘321Walker AVE NE, Huntsville, AL’ and reported it as a ‘good match’ at the address level,which could be misleading. Moreover, we found that approximately 10% of the ‘goodmatch’ rates have been corrected by Google and MapPoint. Yahoo! and MapQuest, on theother hand, give more details on changes to the submitted addresses. Thus, the ‘goodmatches’ by Yahoo! and MapQuest are the coordinates of the actual addresses submitted.The ‘ambiguous match’ rates of Google, MapPoint, and Geocoder.us are less thanMapQuestand Yahoo! because MapQuest and Yahoo! provide an upper level of the hierarchy match tomake sure that there are coordinates returned to the user. Note that ambiguous andunmatched rates for each service are fairly high. We believed this could be due to lack ofzip code information in the submitted addresses. This issue could raise ambiguities, parti-cularly, in the areas with repeated street names. Nonetheless, we found no significantdifference among Google, MapPoint, and Yahoo! on the quality of overall match rates.

Further analysis revealed that different types of location setting and land use are relatedto variability of match rates. The addresses in urban and central city areas were geocodedwith the high percentages of ‘good match’ rates, over 80% by four services (all exceptGeocoder.us), and with the low percentages of ‘ambiguous match’ rates, less than 20% forall five services. ‘Good match’ rates of addresses in suburban areas are slightly lower thanthose in urban areas for all services, consequently increasing their ambiguous andunmatched rates. The addresses in rural setting were geocoded with lower ‘good match’rates (about 40–80%) and higher ‘ambiguous match’ rates (about 10–50%). Assuming thateach geocoding service applies the same underlying geocoding process to all US addresses,the causes of differences between the match rates in urban and rural areas are due to thequality of the reference databases. The street databases in urban and suburban areas arepossibly updated more frequently than those in rural or remote areas. Classified by land use,the residential, commercial, and industrial addresses were geocoded with rather highpercentages of ‘good matches’ and low percentages of ‘ambiguous matches’ for fourservices (all except Geocoder.us). The agricultural addresses are more problematic,

Figure 8. The log-transformed pairwise distances among the services, Geocoder.us (Gus), Google(G), MapPoint (MP), MapQuest (MQ), and Yahoo! (Y), for 386 addresses.


especially for MapQuest and Geocoder.us, than the other address types. Nevertheless, wefound no significant difference among four geocoding services (all except Geocoder.us) foreach land use type.

As for positional accuracy, only about 30% of the geocoded points have error distancesof less than 100 m, as shown in Figure 4. After eliminating outliers, the positional accuracyof each service represented by the medians of error distances ranges from 171 to 238 m. Thisoverall positional accuracy is rather low since it includes all address types. In addition, thereare several factors that could cause the geocoded points to deviate from the baseline, such asthe street offset (the perpendicular distance from the street to the geocoded point) defined byeach geocoding service, and include incomplete, incorrect, and/or out-of-date street data-bases. Another concern is that the baseline is not very accurate. Although EPA reported thattheir sites’ coordinates are within 5 m, this does not guarantee that the points will be at thecentroid of the parcels. This is another hidden factor that influences high deviation ofpositional accuracy. However, since we compared the five services based on ranks withinaddresses, the inferences on the comparison of the positional accuracy are still valid.

We filtered out all addresses that could not be geocoded by all five services in order toperform comparative tests on positional accuracy. It is worth noting that only 70% of allsubmitted addresses could be geocoded by these services. Furthermore, by examiningdifferent address types, we found that the five services generated geocoded results for75% (urban), 77% (suburban), and 52% (rural) addresses, and 75% (residential), 75%(commercial), 47% (agricultural), and 67% (industrial) addresses. The geocoding servicesare likely to produce more unmatched results of the rural and agricultural addresses than ofother address types. This finding is consistent with the match rates we discovered previously.

Regardless of address types and quality types of matches, we found that on average theerror distances generated by Yahoo! (160 m), MapPoint (158 m), and Google (163 m) aresignificantly lower than that of Geocoder.us (194 m) and MapQuest (199 m). Even thoughGoogle and MapPoint introduced a point geocoding technique, known as ‘rooftop’ geocod-ing to improve accuracy, it is worth noting that they do not surpass Yahoo! on positionalaccuracy. MapQuest and Geocoder.us provide a similar level of positional accuracy sinceboth services use the US TIGER data files. For the urban addresses, Google (129 m) andMapPoint (134 m) have slightly lower error distances than Yahoo! (142 m). This could bedue to the types of data in reference databases used by rooftop geocoding. On the other hand,there is no significant difference among geocoding services for the rural addresses that havean average accuracy of about 300–350 m. Another analysis was based on four types of landuse. For the residential addresses, Yahoo!, Google, andMapPoint (,150–159 m) on averagegave a significantly better level of positional accuracy than MapQuest (186 m), whereasGeocoder.us produced the geocoded results at a 168-m accuracy level. These positionalaccuracies agree with the positional accuracy range from 25 to 168 m found in severalstudies, collected and reported by Zandbergen (2008). Among the four types, the geocodedresults of the agricultural addresses are less accurate than any other types, about 300–400 maccuracy similar to that of the rural addresses.

Analyzing the positional accuracy results by considering the quality of matches (goodand ambiguous) and ignoring address types, we found that, generally, good geocoded resultsby all the five geocoding services are comparable at about 163 m accuracy. In general, wecould use the five services interchangeably with minor differences on positional accuracy aslong as the results are identified as ‘good matches’. On the other hand, one has to be carefulwhen using the result of ‘ambiguous matches’ since its positional accuracy may varyconsiderably; in particular our study showed that Google produced the widest range. Ofthe five services, MapPoint provides the least variability of positional accuracy on


‘ambiguous matches’. This could be caused by the high match score at 0.85 set by default forambiguous results.

A closer examination of the relationships among the five geocoding services reveals thesimilarity of the geocoded points using relative distances. These similarity tests confirm theresults we found earlier on positional accuracy and strengthen our inferences on the overallcomparisons. From Figure 8, three pairs of services showed strong similarities by consideringthe medians of pairwise distances: Geocoder.us and MapQuest, Google and MapPoint, andMapPoint and Yahoo!. Geocoder.us and MapQuest showed the strongest similarity with theshortest pairwise distances, partially due to the use of the US TIGER data files by bothservices. For about 50% of the geocoded results, Geocoder.us generated identical points toMapQuest, especially for addresses in urban and suburban areas and for the residential andcommercial addresses. In addition to the US TIGER data files, MapQuest also uses the USGazetteers. This may explain some differences of geocoding results on the same addressesproduced by the two services. As found previously, these two services generally providegeocoded coordinates with less accuracy. Google and MapPoint revealed the second strongestsimilarity with a wide range of variability. This similarity may be introduced by the same datasource for rooftop geocoding that the two providers apply. Strong similarities betweenaddresses within 20 m (50% confidence) in urban and suburban areas and of residential andcommercial addresses can be observed. Greater variations of pairwise distances for these twoservices were found in rural and agricultural addresses.MapPoint and Yahoo! showed the thirdstrongest similarity with a small variation. We found no strong evidence on the referencedatabase or the underlying geocoding processes to further explain why these services producedclosely related results.

7. Conclusions

With the qualification that geocoding is an imperfect process, it is important for developersand researchers to realize the quality of geocoding results before utilizing a geocodingservice in their applications or projects. This article has presented an empirical comparisonof five online geocoding services offered by Geocoder.us, Google, MapPoint, MapQuest,and Yahoo!. Only US addresses were used in this study and geocoded results were comparedagainst the EPA baseline data. The geocoding quality was determined by match rates,positional accuracy, and similarity.

The results reveal several differences among these five services. In general, Geocoder.usproduced lower match rates than other services. As we further classified the quality of matchrates as good, ambiguous, and unmatched, we found that, overall, the services can beseparated into three groups. The first group having high good match rates consists ofGoogle, MapPoint, and Yahoo!. The second group having moderate quality with high‘good matches’ and high ‘ambiguous matches’ is MapQuest. The third group with highunmatched rates is Geocoder.us. Considering together with the impact of address types,Geocoder.us provided the lowest quality of match rates for all address types while the firstand the second group provided roughly similar quality of match rates for urban, rural, andindustrial addresses.

Yahoo!, MapPoint, and Google, on average, provided more accurate points and shortererror distance, than Geocoder.us and MapQuest. However, all services equally generatedless accurate results for rural, agricultural, and industrial addresses. We further investigatedthe positional accuracy of the results marked as good and ambiguous matches by eachservice’s report. The accuracy of ‘good matches’ is approximately equivalent for all services


(about 163 m). On the contrary, the accuracy of ‘ambiguous matches’ varies considerablywithin and between services.

The pairwise distance was used to reflect similarity among services. We found three pairsof similar services, generating nearby geocoding points: Geocoder.us and MapQuest,Google and MapPoint, and MapPoint and Yahoo!.

Notes1. http://googlemapsapi.blogspot.com/2008/05/shout-it-from-rooftops.html2. http://blogs.msdn.com/mappoint_b2b/archive/2007/04/17/rooftop-geocoding-now-

available.aspx3. http://www.epa.gov/ttn/airs/airsaqs/detaildata/downloadaqsdata.htm

References

Antenucci, J.C., et al., 1991. Geographic information systems: a guide to the technology. New York:Van Nostrand Reinhold.

Bonner, M.R., et al., 2003. Positional accuracy of geocoded addresses in epidemiologic research.Epidemiology, 14, 408–412.

Cayo, M.R. and Talbot, T.O., 2003. Positional error in automated geocoding of residential addresses.International Journal of Health Geographics, 2, 1–12.

Christen, P., Churches, T. andWillmore, A., 2004. A probabilistic geocoding system based on a nationaladdress file. In: Proceedings of the 3rd Australasian data mining conference, Cairns, Australia.

Churches, T., et al., 2002. Preparation of name and address data for record linkage using hiddenMarkov models. BMC Medical Informatics and Decision Making, 2, 1–16.

Drummond, W.J., 1995. Address matching: GIS technology for mapping human activity patterns.Journal of the American Planning Association, 61, 240–251.

Goldberg, D.W.,Wilson, J.P. andKnoblock, C.A., 2007. From text to geographic coordinates: the currentstate of geocoding. Journal of the Urban and Regional Information Systems Association, 19, 33–46.

Gu, L., et al., 2003. Record linkage: current practice and future directions. Report 03/83 CSIROMathematical and Information Sciences, Australia.

Huxhold, W.E., 1991. An introduction to geographic information systems. Oxford: Oxford University Press.Karimi, H.A., Durcik, M., and Rasdorf, W., 2004. Evaluation of uncertainties associated with geocod-

ing techniques. Computer-Aided Civil and Infrastructure Engineering, 19, 170–185.Krieger, N., 2001. On the wrong side of the tracts? Evaluating the accuracy of geocoding in public

health research. American Journal of Public Health, 91, 1114–1116.Longley, P., et al., 2005. Geographic information systems and science. New York: John Wiley and

Sons.Lovasi, G.S., et al., 2007. Comparing a single-stage geocoding method to a multi-stage geocoding

method: how much and where do they disagree? International Journal of Health Geographics, 6,1–11.

Strickland, M.J., et al., 2007. Quantifying geocode location error using GIS methods. EnvironmentalHealth, 6, 10–17.

Ward, M.H., et al., 2005. Positional accuracy of two methods of geocoding. Epidemiology, 16,542–547.

Whitsel, E.A., et al., 2004. Accuracy and repeatability of commercial geocoding. American Journal ofEpidemiology, 160, 1023–1029.

Whitsel, E.A., et al., 2006. Accuracy of commercial geocoding: assessment and implications.Epidemiologic Perspectives & Innovations, 3, 1–12.

Yang, D.-H., et al., 2004. Improving geocoding practice: evaluation of geocidng tools. Journal ofMedical Systems, 28, 361–370.

Zandbergen, P.A., 2008. A comparison of address point, parcel and street geocoding techniques.Computers, Environment and Urban Systems, 32, 214–232.

Zandbergen, P.A. and Green, J.W., 2007. Error and bias in determining exposure potential of childrenat school locations using proximity-based GIS techniques. Environmental Health Perspectives,115, 1363–1370.


http://googlemapsapi.blogspot.com/2008/05/shout-it-from-rooftops.html

http://blogs.msdn.com/mappoint_b2b/archive/2007/04/17/rooftop-geocoding-now-available.aspx



http://www.epa.gov/ttn/airs/airsaqs/detaildata/downloadaqsdata.htm

Copyright of International Journal of Geographical Information Science is the property of Taylor & Francis Ltd

and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright

holder's express written permission. However, users may print, download, or email articles for individual use.

Documents

Comparative evaluation and analysis of online geocoding services