Upload
hector-jacobs
View
222
Download
1
Embed Size (px)
Citation preview
Introduction to Web APIs Tianjun [email protected] 2011
Google Custom Search, Amazon, eBay, Flickr, YouTube, Second Life
1
2
Web APIs
Application Programming Interface (API) a particular set of rules and specifications that a software
program can follow to access and make use of the services and resources provided by another particular software program that implements that API.
serves as an interface between different software programs and facilitates their interaction
Web API typically a defined set of HTTP request messages expressed
in SOAP or REST along with a definition of the structure of response messages, typically expressed in JSON or XML.
3
SOAP
Simple Object Access Protocol (SOAP)
SOAP Tutorial: http://www.w3schools.com/soap/soap_intro.asp
A SOAP message is an ordinary XML document containing the following elements: An Envelope element that identifies the XML document as a SOAP
message A Header element that contains header information▪ contains application-specific information about the SOAP message▪ optional▪ must be the first child element of the Envelope element
A Body element that contains call and response information A Fault element containing errors and status information
4
REST
Representational State Transfer (REST) Use HTTP method to invoke remote services (not XML)
The response of remote service can be in XML or any textual format
Benefits: Easy to develop Easy to debug (with standard browser) Leverage existing web application infrastructure
We will focus on REST services programming for the rest of the slides.
5
Server Responses
Really Simple Syndication (RSS, Atom) XML-based standard Designed for news-oriented websites to “Push” content to
readers Excellent to monitor new content from websites
JavaScript Object Notation (JSON) Lightweight data-interchange format Human readable and writable and also machine friendly Wide support from most languages (Java, C, C#, PHP,
Ruby, Python…)
JSON Example
http://www.json.org/example.html
7
Google API
Google has many APIs available: http://code.google.com/more/
We will introduce two of them: Google Custom Search API YouTube Data APIs
8
Google Custom Search
What is custom search? to create a search engine tailored to your needs and
interests Your custom search engine (CSE) can prioritize or
restrict search results based on websites you specify
you can present the results in your website
9
Create your CSE
Developer's Guide http://code.google.com/apis/customsearch/docs/dev_guide.html
You can use a provided custom search wizard to create your own CSE: http://www.google.com/coop/manage/cse/create/1
You can use the “control panel” to manage your CSE. You can access the “control panel” from the “My search engines” page. http://code.google.com/apis/customsearch/docs/start.html#managing
10
What’s in a CSE
A custom search engine has two main components housed in XML files: Context: specifies the global settings of the search
engine. Each CSE has its own context file. Annotations: lists the sites (webpages or websites)
you want your search engine to cover. You can view or download these two XML
files in “control panel”->”Advanced”.
11
Context and Annotation Files:
Your CSE ID and your personer ID
12
Access your CSE
Users can access your CSE in four ways: Your custom search engine homepage A search engine gadget in their iGoogle pages A search engine gadget in their webpages or blogs A search box in your website▪ To get your very own search box, copy the generated
code from the “Code” tab of the control panel, and paste it into the HTML of your webpage.
http://code.google.com/apis/customsearch/docs/start.html#release
13
Google Custom Search API
The Custom Search API lets you skip the control panel altogether and use a client application of your choice to : retrieve a list of CSE under an account Create, update and delete custom search engines retrieve code for a search box retrieve current settings of a CSE associate a CSE with an AdSense Account http://code.google.com/apis/customsearch/docs/api.html
14
Request and Response Format
Request Format: standard HTTP requests, such as: GET, POST, and
DELETE. Need to get an authentication token from ClientLogin
using HTTP post Request▪ Generally an authentication token expires in two weeks▪ http://code.google.com/apis/accounts/docs/AuthForInstalledAp
ps.html Response Format:
XML-based Format Details:
http://code.google.com/apis/customsearch/docs/api.html
15
Request Example
A POST request that create a search engine specification (by submitting a context xml).
authentication token
New context file content
Request Endpoint
16
Access CSE Search Results
XML feeds of search results are only available to paid Business account …
Our Method 1: Parse the content of the search result page:▪ http://www.google.com/cse?cx=creatorId:SEid&q=
query¶meter=value▪ creatorID and Seid can be found in context xml file of your CSE▪ Parameter and query details can be found in
http://code.google.com/apis/customsearch/docs/structured_search.html
17
Access CSE Search Results
Method 2: Google Custom Search Element for JavaScript users▪ The generated code from the control panel use this method▪ http://code.google.com/apis/customsearch/docs/js/cselement-devguide.html
Method 3: Use JSON/Atom Custom Search API (RESTful requests)to get
search results in either JSON or Atom format.▪ http://code.google.com/apis/customsearch/v1/overview.html
Method 2 and 3 need to visit the Google APIs console to set up a new project and get an API key. Currently an API key allows 100 queries per day and 100 results at
most for each query. You can request additional quota from its console.
18
Summary for Google CSE API
Step 1: create your own CSE using the wizard (web interface) or the CSE API CSE API needs to apply for authentication token
Step 2: manage your CSE using the wizard or the CSE API if necessary
Step 3: access CSE search results by method 1,2,3. Method 2 & 3 needs to apply for API key
19
Amazon
Amazon Web Services (AWS) http://aws.amazon.com/ “With AWS you can requisition compute power, storage, and other services
…” Amazon SDK for Java
http://aws.amazon.com/sdkforjava/ Amazon Product Advertising API
https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html
Access Amazon Product Selection▪ gives you access to Amazon’s selection of millions of products in categories such as
books, music … Leverage Amazon Product Discovery capabilities▪ lets you leverage Amazon’s customer-centric features such as Product Search,
Customer Reviews, Similar Products... Monetize your website
20
Amazon Product Advertising API
Programming Guide http://docs.amazonwebservices.com/AWSECommerceService/latest/D
G/index.html?ProgrammingGuide.html Check “Visual Introduction to Product Advertising API” to
see what this API can do. E.g.: to find items
21
Amazon Product Advertising API: Request Format
Tutorial: http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/index.
html?CHAP_MakingRequestsandUnderstandingResponses.html Format: REST & SOAP REST Format:
http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&
Operation=ItemSearch& AWSAccessKeyId=[Access Key ID]&AssociateTag=[ID]&
SearchIndex=Apparel& Keywords=Shirt&Timestamp=[YYYY-MM-DDThh:mm:ssZ]
&Signature=[Request Signature]
22
Amazon Product Advertising API: Request Authentication
Tutorial: http://docs.amazonwebservices.com/AWSECommerceServic
e/latest/DG/index.html?RequestAuthenticationArticle.html
To access the Product Advertising API, you must first create an account at http://aws.amazon.com. AWS will assign you a pair of related identifiers: Access Key ID (a 20-character, alphanumeric sequence) ▪ include it in AWS service requests to identify yourself as the sender
of the request Secret Access Key (a 40-character sequence)▪ To calculate the signature used in your request to verify your identity
23
HMAC-SHA Signatures for REST Requests Authentication
Two authentication parameters: Timestamp: ▪ a dateTime object, with the complete date plus hours,
minutes, and seconds▪ Format : YYYY-MM-DDThh:mm:ssZ (where T and Z
are literals) Signature:▪ created by using the request type, domain, the URI, and a
sorted string of every parameter in the request ▪ Once properly formatted, you create a base64-encoded
HMAC_SHA256 signature using your AWS secret key
24
HMAC-SHA Signatures for REST Requests Authentication (cont.)
Tutorial to sign a request, using sample AWS access identifiers (READ IT!). http://docs.amazonwebservices.com/AWSECommerceService/latest/D
G/index.html?rest-signature.html Basic Authentication Process
25
Amazon Product Advertising API: Response
Response is a XML file which conforms to a WSDL and schema.
WSDL location: http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/i
ndex.html?WSDLLocation.html
Paging Through Results http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/i
ndex.html?PagingThroughResults.html E.g., for itemsearch request, use “&ItemPage=4 ” to access fourth page of
the results.
Sorting Results http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/i
ndex.html?SortingResults.html E.g., for itemsearch request, use “&Sort=titlerank” to sort results by title
or description
26
Amazon Product Advertising API:Steps to search Amazon
Steps to search Amazon: Apply for a Access Key ID and Secret Access Key Generate search conditions Generate a parameter string Generate a string used for signature Generate a signature Generate a query URL Send a search request to Amazon server Receive the query result (XML format) Parse the result and display it on your web site
27
eBay API
Main page: http://developer.ebay.com/common/api/
eBay Web Services Overview http://developer.ebay.com/products/overview/ Available APIs include:▪ Finding API, Best Match API, Merchandising API▪ Shopping API, Feedback API, Trading API, Client Alerts API ▪ Platform Notifications API, Research API for eBay, etc.
Which API to use with your application?▪ http://developer.ebay.com/products/overview/api-by-feature/
28
eBay Finding API
eBay Finding API: to search for eBay items: http://developer.ebay.com/products/finding/ Search for items based on a keyword query Search for items associated with a specific product Browse for items in a specific category or in an eBay store Filter search results by item listing details, such as price, condition, etc. Refine searches by the characteristics of an item (i.e., aspects), such as brand, style, etc. Retrieve category and aspect metadata Get keyword recommendations to improve search results
Formats: XML, SOAP, Name Value, JSON
Protocols: HTTP GET and POST
Need to apply for a API key https://developer.ebay.com/join/default.aspx
29
eBay Finding API: Call Structure
Tutorial for making a API call: http://developer.ebay.com/DevZone/finding/Concepts/MakingACall.html
Each Finding API call consists of the following elements: Service Endpoint HTTP Headers or URL Standard Input Fields Call-specific Fields
URL-based call structure http://svcs.ebay.com/services/search/FindingService/v1?
OPERATION-NAME=findItemsByKeywords
&SERVICE-NAME=FindingService
&SERVICE-VERSION=1.0.0
&GLOBAL-ID=EBAY-US &SECURITY-APPNAME=MyAppID
&RESPONSE-DATA-FORMAT=XML
&REST-PAYLOAD
&Standard input fields
&Call-specific input fields
30
eBay Finding API: findItemsByKeywords
findItemsByKeywords lets you to search for items on eBay by a keyword query http://developer.ebay.com/DevZone/finding/CallRef/findItemsByKeywords.html
Controlling Results with Item Filters▪ &itemFilter(0).name=FreeShippingOnly&itemFilter(0).value=true
&itemFilter(1).name=ListingType&itemFilter(1).value=FixedPrice Refining Results with Aspect and Domain Filters▪ &aspectFilter.aspectName=Sub Style&aspectFilter.aspectValueName=Running
Shoes...▪ &domainFilter.domainName=Digital_Cameras
Finding Local Items▪ &buyerPostalCode=95125&sortOrder=Distance
Sorting the Output: ▪ &sortorder= PricePlusShippingAsc
Paginating the Results▪ &paginationInput.pageNumber=2&paginationInput.entriesPerPage=50
31
eBay Finding API: findItemsByKeywords Examples
Call Example: The user is looking for a specific Harry Potter title and has a maximum price ceiling and doesn't want to pay for shipping. http://svcs.ebay.com/services/search/FindingService/v1?
OPERATION-NAME=findItemsByKeywords& SERVICE-VERSION=1.9.0& SECURITY-APPNAME=YourAppId& RESPONSE-DATA-FORMAT=XML& REST-PAYLOAD& keywords=harry%20potter%20phoenix& itemFilter(0).name=MaxPrice& itemFilter(0).value=10.00& itemFilter(0).paramName=Currency& itemFilter(0).paramValue=USD& itemFilter(1).name=FreeShippingOnly& itemFilter(1).value=true& paginationInput.entriesPerPage=2
PHP coding Example and Tutorial: http://developer.ebay.com/DevZone/finding/HowTo/GettingStarted_PHP_NV_XML/GettingStarted_PHP_NV_XML.html
32
eBay Finding API: Steps to search eBay
Steps to search eBay: Apply for a API key Construct the request Send the request to eBay Server Receive the query result in your preferred format Parse the result
33
Flickr API: General
Flickr API Main Page: http://www.flickr.com/services/api/
API Key Application http://www.flickr.com/services/api/keys/
Request Format Rest, XML_RPC, SOAP
Response Format Rest, XML_RPC, SOAP, JSON, PHP
34
Flickr API: Photo Search
Flickr Photo Search Method Description: http://www.flickr.com/services/api/flickr.photos.search.html
Explorer: http://www.flickr.com/services/api/explore/?method=flickr.photos.search
Steps: Apply for a Flickr API Key Send a search request to Flickr Server in your preferred
format Receive the response in your preferred format Parse the result and get the photo URLs▪ To construct photo URLs:
http://www.flickr.com/services/api/misc.urls.html
35
Flickr API: Photo Search Example – REST format
Request Format: http://api.flickr.com/services/rest/?
method=flickr.photos.search&api_key=yourkey&argument=value Example:
Request (to search photos that have “tucson” in its description): http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=e7c88ea43041aff2df564188d4609849&text=tucson
Response:
Extracted Image URL and Web Page URL:▪ http://farm6.static.flickr.com/5288/5368668124_d216292c15.jpg▪ http://www.flickr.com/photos/48553200@N04/5368668124
36
YouTube API
YouTube Data API: lets you incorporate YouTube functionality into your own
application or website. You can perform searches, upload videos, create playlists, and more.
Main Page: http://code.google.com/apis/youtube/getting_started.html#data_api
Java Developer Guide: http://code.google.com/apis/youtube/2.0/developers_guide_java.html
YouTube Player API give you control over YouTube video playback on your
website.
37
YouTube Data API
Advanced functions of the data API requires its java client library http://code.google.com/apis/youtube/2.0/developers_guide_java.html#Getting_
Started
You will need to apply for a developer key and client id to execute authenticated functions. http://code.google.com/apis/youtube/2.0/developers_guide_java.html#Authenti
cation
Two important class: VideoFeed: represent lists of videos, such as standard feeds, uploads,
subscriptions, and favorite videos VideoEntry: Each video entry corresponds to exactly one YouTube video and
contains information about that video.
38
YouTube Video Search
Description of the YouTube video search function http://code.google.com/intl/en/apis/youtube/2.0/developers_guide_java.html#Searching_for_Videos
Query parameter definition http://code.google.com/intl/en/apis/youtube/2.0/reference.html#Query_parameter_definitions
A search query example for videos that match the search term “puppy”. http://gdata.youtube.com/feeds/api/videos?q=puppy&orderby=viewCount
The query example can be generated using the following code: YouTubeQuery query =
new YouTubeQuery(new URL("http://gdata.youtube.com/feeds/api/videos")); query.setOrderBy(YouTubeQuery.OrderBy.VIEW_COUNT); query.setFullTextQuery("puppy"); //print resultVideoFeed videoFeed = service.query(query, VideoFeed.class); printVideoFeed (videoFeed, true);
39
YouTube Video Search Steps
Steps to search YouTube Videos: Download client library Apply for Developer key and client id Construct a search request and send to YouTube Receive the query result Parse the result and display videos on your
websites.
40
Second Life API
Second Life API Main Page: http://wiki.secondlife.com/wiki/Web_Services_Portal
There are eight official APIs: Registration API Map API Exchange Risk API Media Plugin API Live Data Feeds Inventory API Snapshot API Search API
41
Second Life API: Search API
The Second Life Search API enables you to search Second Life for: Events, People, Places, Groups, and information on
its Wiki. http://wiki.secondlife.com/wiki/SearchAPI
Linden Lab uses a Google Search Appliance to index web pages. The Search API allows to search the indexed web pages. No API key is needed.
42
Second Life API: Search API Request Format
Request format: http://search.secondlife.com/client_search.php?
q=yourquery¶emeter=value Two popular parameters:
start▪ Each query will return 20 results at most (wiki says 10)▪ “start=0” shows the first 20 results, “start=20” shows 21-40, etc.
s▪ To limit the results to specific categories▪ This value defaults to "All", but can also be set to one of the
following: All, Events, Groups, People, Places, Wiki.
43
Second Life API: Search API Response Format
The response format is XHTML. Example:
http://search.secondlife.com/client_search.php?q=arizona&s=All&start=20
44
Second Life API: Search API Steps
Steps: Send a search request to Second Life Server Parse the XHTML response page to get each result
page URL. Parse each result page to get information:▪ Pages in each category share similar structure. You can
develop separate web page parsers for group page, event page, place page, etc.
45
XML Processing
A variety of APIs for accessing XML have been developed and used, and some have been standardized. Existing APIs for XML processing tend to fall into these categories: Stream-oriented APIs accessible from a programming language▪ E.g., Simple API for XML (SAX)
Tree-traversal APIs accessible from a programming language.▪ E.g., Document Object Model (DOM)
XML data binding, which provides an automated translation between an XML document and programming-language objects▪ E.g., Java Architecture for XML Binding (JAXB)
Declarative transformation languages▪ E.g., XSLT, XPath and XQuery.▪ XPath Tutorial: http://www.w3schools.com/xpath/default.asp
46
DOM
Document Object Model (DOM) DOM accesses an XML document through a tree
structure, composed of element nodes and text nodes.
47
DOM Example
<a href="http://www.google.com/">Google's<b>fast</b> home page.</a>
48
DOM
Pros of DOM: Easy to manipulate the document Can traverse the document back and forth Good for small XML files
Cons of DOM: Consumes lots of memory
JDK provides two packages related to DOM: org.w3c.dom javax.xml.parsers
Some popular JAVA xml parsers: Xerces JDOM DOM4j
49
JSON Processing
JSON Tutorial: http://www.json.org/
A lot of developed JSON tools and parsers are listed in the website. Listed by different language
For example: http://www.json.org/java/