Upload
dag-endresen
View
352
Download
1
Tags:
Embed Size (px)
Citation preview
GBIF & Species2000 Hackathon March 2015 Naturalis, Leiden
GBIF portal API
Dag Endresen GBIF Norway UiO Natural History Museum in Oslo University of Oslo Tuesday, March 3rd, 2015 Slides: CC-BY-4.0
Credits, some slides from:
Daniel Amariles, Colombia (2013) Nodes training at GBIF GB20 in Berlin [link]
Gallien Labeyrie, France (2014) Mentoring project France-Spain-Portugal [link]
GBIF DATA PORTAL
SPECIES SEARCH
GBIF DATA PORTAL API
An interface to access data published through the GBIF network using web services.
DATA PORTAL API
GBIF Data Portal API: http://api.gbif.org/v1/ (+parameters)
Summary and information:
http://www.gbif.org/developer/summary The RESTful API take search parameters as key=value pairs and respond with json content type.
RESTful query formatJSON response type
GBIF API sections
• Registry informa)on about the datasets, organiza)ons (e.g. data publishers), networks and the means to access them (technical endpoints)
• Species informa)on about species and higher taxa, and u)lity services for interpre)ng names and looking up the iden)fiers (access to all published checklists in the GBIF checklist bank)
• Occurrence occurrence informa)on crawled and indexed by GBIF and search services to do real )me paged search and asynchronous download services to do large batch downloads
• Maps simple services to show the maps of GBIF mobilized content
• News services to stream useful informa)on (RSS)
API example : dataset Search for datasets by publishing country: http://api.gbif.org/v1/dataset/search?publishingCountry=NO
Dataset information (UiO NHM Lichens):http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670-ed1015b26252
Contact persons for a dataset:http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670-ed1015b26252/contact
Dataset endpoint (get download URL): http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670-ed1015b26252/endpoint
http://www.gbif.org/developer/registry
API example : dataset Download activity metrics for dataset (UiO NHM Lichens): http://api.gbif.org/v1/occurrence/download/dataset/7948250c-6958-4a29-a670-ed1015b26252
=> records from this dataset, included in 2650 download sets
Records lastInterpreted since November 2014: http://api.gbif.org/v1/occurrence/search?datasetKey=7948250c-6958-4a29-a670-ed1015b26252&lastInterpreted=2014-11-01,* (=> 168 316 occ.)
Metrics for dataset data contents: http://api.gbif.org/v1/dataset/66dd0960-2d7d-46ee-a491-87b9adcfe7b1/metrics
=> count records by dimensions such as Kingdom, Rank, Vernacular name langue, Extensions provided, … NB! only implemented for species checklists, not (yet?) for occurrences!
http://www.gbif.org/developer/registry
API example : species List all name usages (across all checklists): http://api.gbif.org/v1/species?name=Beta%20vulgaris
Name usage across checklists (Beta vulgaris, 5383920): http://api.gbif.org/v1/species/5383920/related
Name parsed into epithets and author etc.: http://api.gbif.org/v1/parser/name?name=Abies%20alba%20Mill.%20sec.%20Markus%20D.
{"scientificName": "Abies alba Mill. sec. Markus D.", "type": "SCINAME", "genusOrAbove": "Abies", "specificEpithet": "alba", "authorsParsed": true, "authorship": "Mill.", "sensu": "sec. Markus D.", "canonicalName": "Abies alba", "canonicalNameWithMarker": "Abies alba", "canonicalNameComplete": "Abies alba Mill." }
http://www.gbif.org/developer/species
API example : occurrence List occurrences of Beta vulgaris: http://api.gbif.org/v1/species/match?name=Beta+vulgaris => taxonKey http://api.gbif.org/v1/occurrence/search?taxonKey=5383920
List occurrences from Norway (of Beta vulgaris): http://api.gbif.org/v1/occurrence/search?publishingCountry=NO http://api.gbif.org/v1/occurrence/search?publishingCountry=NO&taxonKey=5383920
Information about a single occurrence record: http://api.gbif.org/v1/occurrence/1040970640 http://api.gbif.org/v1/occurrence/1040970640/fragment http://api.gbif.org/v1/occurrence/1040970640/verbatim
List occurrence counts for datasets of country (or taxon):
http://api.gbif.org/v1/occurrence/counts/datasets?country=NO
http://www.gbif.org/developer/occurrence
API example : identifiers (not implemented -- yet)
Searching by occurrenceID is unfortunately not supported yet … http://dev.gbif.org/issues/browse/POR-2451 http://dev.gbif.org/issues/browse/POR-2337 So, we cannot yet list or count occurrences with pattern “urn*” in occurrenceID
http://api.gbif.org/v1/occurrence/search?occurrenceID=urn* http://api.gbif.org/v1/occurrence/search?occurrenceID=urn%3Acatalog http://api.gbif.org/v1/occurrence/search?occurrenceID=urn%3Alsid http://api.gbif.org/v1/occurrence/search?occurrenceID=http* …
API example : download data Lookup speciesKey (1) and download occurrences (2): http://api.gbif.org/v1/species/match?verbose=false&kingdom=Plantae&name=Beta+vulgaris
=> usageKey/speciesKey = 5383920
http://api.gbif.org/v1/occurrence/search?taxonKey=5383920 [&limit=1000&offset=0]
=> notice: count = 25 513 => then: page through results… (using offset & limit)
http://api.gbif.org/v1/occurrence/download/request [POST] => downloadKey (see next slide)
API example : asynchronous (1) Request asynchronous download:
$ curl -i --user yourGbifUserName:yourGbifPassord -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d @filter.json http://api.gbif.org/v1//occurrence/download/request >> log.txt
Search parameters in a json text file: filter.json (in current directory or located in a “PATH-directory”):
{ "creator":”yourGbifUserName", "notification_address": [“[email protected]"], "predicate": { "type":"and", "predicates": [{"type":"equals","key":"HAS_COORDINATE","value":"false"}, {"type":"equals","key":"TAXON_KEY","value":"5383920"}] } }
Downloads are available in the portal (from your user profile)
API example : asynchronous (2a) Request asynchronous download:
function gbifapi { curl -i –user yourGbifUserName:yourGbifPassword -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d "{\"creator\":\”yourGbifUserName\", \"notification_address\": [\”[email protected]\"], \"predicate\": {\"type\":\"and\", \"predicates\": [{\"type\":\"equals\",\"key\":\"HAS_COORDINATE\",\"value\":\"true\"}, {\"type\":\"equals\", \"key\":\"TAXON_KEY\", \"value\":\"$1\"}] }}" http://api.gbif.org/v1/occurrence/download/request >> log.txt echo -e "\r\n$1 $2\r\n\r\n----------------\r\n\r\n" >> log.txt
} $ gbifapi 4140730 "Aciachne acicularis" $ gbifapi 4140704 "Aciachne flagellifera" $ gbifapi 5289784 "Aegilops comosa” … (work in progress…)
API example : asynchronous (2b) (…clean log.txt with the downloadKeys using regular expressions…)
function gbifwget { echo -e "\n\n----------------\n$1 $2 $3\n" >> log_wget.txt wget http://api.gbif.org/v1/occurrence/download/request/$1.zip 2>&1 | tee /dev/tty >> log_wget.txt mv $1.zip ./dwca/$2.zip 2>&1 | tee /dev/tty >> log_wget.txt } $ gbifwget 0006050-141024112412452 4140730 "Aciachne acicularis" $ gbifwget 0006053-141024112412452 4140704 "Aciachne flagellifera" $ gbifwget 0006056-141024112412452 5289784 "Aegilops comosa" … (work in progress…)
MAPPING API v1.0 You can easily overlay GBIF content on your own maps. http://www.gbif.org/developer/maps
Slide by Daniel Amariles, 2013
This service is intended for use with commonly used map clients such as the Google Maps API, Leaflet JS library or Modest maps JS library. These libraries allow the GBIF layers to be visualized with other content, such as those coming from Web Map Service (WMS) providers. It should be noted that the mapping API is not a WMS service, nor does it support WFS capabili)es.
hNp://leafletjs.com/
MAPPING API v1.0
hNp://modestmaps.com/
Slide by Daniel Amariles, 2013
CUSTOMIZING LAYER CONTENT
The format of the URL is as follows: With the following required parameters:
type : TAXON, DATASET, COUNTRY or PUBLISHER
key : The appropriate key for the chosen type (a taxon key, dataset/publisher UUID or 2 leNer ISO country code)
Other supported parameters: resolu)on, layer, paleNe, colors, satura)on, hue
hNp://www.gbif.org/developer/maps
hNp://api.gbif.org/v1/map/density/)le?x={x}&y={y}&z={z}
Slide by Daniel Amariles, 2013
Useful Tools (JSON & REST)
• REST client … • JSON client/parser …
• JSONView (Firefox, Chrome, …)
• http://jsonview.com/ • Display formatted JSON in browser
• R CRAN : jsonlite • http://cran.r-project.org/web/packages/jsonlite/
• E.g. read json into a dataframe [link]
• OpenRefine • http://openrefine.org/
R CRAN
rOpenSci provides programmatic access to scientific data with R (rgbif, taxize, EML, geonames, …). https://github.com/ropensci http://ropensci.org/packages/ http://ropensci.org/tutorials/rgbif_tutorial.html http://ropensci.org/tutorials/taxize_tutorial.html
rOpenSci : rgbif library(rgbif) key <- name_backbone(name='Beta vulgaris', kingdom=‘Plantae')$speciesKey bv <- occ_search(taxonKey=key, return='data', hasCoordinate=TRUE, limit=1000) gbifmap(bv)
raster : WorldClim, BioClim layers # using GBIF data (bv) from the previous slide… library(raster) xy <- cbind('lon'=bv$decimalLongitude, 'lat'=bv$decimalLatitude); env <- getData('worldclim', var='bio', res=10) # bioclim (pkg raster) plot(env, 1) # plot the first bioclim layer points(xy[,'lon'], xy[,'lat'], col='red') # plot points bio <- extract(env, xy); # extract environment to points (pkg raster) bv_bio <- cbind(bv, bio); # column-bind GBIF-data and bioclim
rOpenSci : rWBclimate library(rWBclimate, ggplot2) country_dat <- get_historical_temp(c("NOR", "SWE", "DNK", "FIN"), "year") ggplot(country_dat, aes(x = year, y = data, group = locator)) + theme_bw(base_size=18) + geom_point() + geom_path() + labs(y="Average annual temperature of Nordic countries", x="Year") + stat_smooth(se = F, colour = "black") + facet_wrap(~locator, scale = "free")
Resolve taxonomic names library(taxize) # rOpenSci Taxize gnr <- gnr_resolve(names = "Beta vuulgariss") # Misspelled name gnr$results # display suggested names submitted_name matched_name data_source_title score 1 Beta vuulgariss Beta vulgaris L. Catalogue of Life 0.75 2 Beta vuulgariss Beta vulgaris L. ITIS 0.75 3 Beta vuulgariss Beta vulgaris NCBI 0.75 4 Beta vuulgariss Beta vulgaris var.-gr. crassa Alef. GRIN Taxonomy for Plants 0.75
specieslist <- c("Beta vulgaris", "Phleum pratensis", "Nicotiana glauca") classification(specieslist, db = 'itis') # lookup higher taxonomy
Global Names Resolver: http://resolver.globalnames.org/ rOpenSci Taxize: http://ropensci.org/tutorials/taxize_tutorial.html
db = ’col' db = ’itis'
rOpenSci : EML
library(EML, rfigshare) description <- "My dataset published in GBIF"
eml_write(dat = dat, meta, title = "My Dataset", description = description, creator = "Your Name <[email protected]>", file = "dataset.xml") eml_publish("dataset.xml", description = description, categories = "Ecology", tags = "biodiversity", destination = "figshare", visibility = "public") meta <- eml_read("eml_example.xml")
GBIF Hackathon March 2015 Naturalis, Leiden
GBIF portal API
Dag Endresen GBIF Norway UiO Natural History Museum in Oslo University of Oslo Tuesday, March 3rd, 2015 Slides: CC-BY-4.0