Web Communication Using MATLAB and VBA

Kipp Martin

March 1, 2012

1

Lecture Files

Files for this module:

I retailerSQL.m

I mortgageWebService.m

I parseSqlXML.m

I http://gsbkip.chicagogsb.edu/xml/dooWopQuery.xml

2

Outline

Motivation

Using the Browser

Web Services

Web Services in MATLAB

XML Background

XML and MATALB

XML and DOM

3

Motivation

Motivation: Getting data from external sources!

Data that is needed for a model may reside in text files (done).

Data that is needed for a model may reside in other worksheetsand workbooks (done).

In this module, we get data from the Web.

4

Using the Browser

Basic concept: the browser communicates with a Web Server.

5

Using the Browser

The browser consists of:

I User Interface/GUI

I Libraries/Routines for:

I rendering HTML

I a Javascript interpreter

I routines for HTTP protocol communication

6

Using the Browser

Do the following:

I open your browser

I type in the url

http://faculty.chicagobooth.edu/kipp.martin/root/

martin.html

I View the “source” – you will see html that was downloadedby the browser and formatted for your viewing pleasure

7

Using the Browser

Now let’s try something a bit more interesting

I open your browser

I type in the url

http://ichart.finance.yahoo.com/table.csv?s=

INTC&a=00&b=1&c=2000

I Either save the file or view it, depending on your browser

8

Using the Browser

Big idea: do what we just did through a browser inside MATLABor VBA (we can just strip off the user interface)

I MATLAB – use the method urlread( url)

I VBA – use

Dim myBrowser As Object

Set myBrowser = CreateObject("InternetExplorer.Application")

myBrowser.Navigate "url"

In the MATLAB command window type

urlread(’http://faculty.chicagobooth.edu/kipp.martin/root/martin.html’)

urlread(’http://ichart.finance.yahoo.com/table.csv?s=INTC&a=00&b=1&c=2000’)

9

Using the Browser

Next big idea: an application program interface (API) – a set orrules or a protocol for interacting with the Web server.

Pass parameters to the server in the URL!

Here is the API for Yahoo finance (see, for example,http://etraderzone.com/free-scripts/

47-historical-quotes-yahoo.html)

http://ichart.finance.yahoo.com/table.csv?s=tic&a=

06&b=9&c=1986&d=2&e=5&f=2008&g=d

I tic – the ticker symbol

I a – input of start month (00 for January, etc.)

10

Using the Browser

I b – start day

I c – start year

I d – end month

I e – end day

I f – end year

I g – time interval (’d’, ’w’, ’m’)

11

In Class Exercise

Write a MTLAB function getYahooAvgHigh( ticker ) that takesthe argument ticker that is the ticker symbol and returns themean high for that stock.

A few hints:

I Create a string yahooURL that is the URL that goes to theserver

I The result of urlread( yahooURL) is a ???? – you betterknow the answer to this?

I Use strread to read the character array

[a,b,c] = strread(s,’%s%d%d’,’delimiter’,’,’)

12

In Class Exercise

A few hints continued:

I strread will recursively read through the entire string array

I in

[a,b,c] = strread(s,’%s%d%d’,’delimiter’,’,’)

[a, b, c] is a cell array

I in our case we have

[date, open, high, low, close, volume, adj_close]

13

Using the Browser

Second Example: Google Maps!

Type the URL

http://maps.googleapis.com/maps/api/distancematrix/

xml?origins=Chicago&destinations=Cincinnati&mode=

driving&sensor=false

into your browser.

Next use as argument to urlread.

See http://code.google.com/apis/maps/documentation/

distancematrix/

14

Using the Browser

Two big improvements:

I A better API – Web Services

I Better parsing – xml

15

Web Services

Key Idea: Have two or more programs talk to each other over anetwork.

In our case, these programs are MATLAB and Excel.

We use MATALB and VBA to orchestrate the communication.

16

Web Services

GUI Based Communication

17

Web Services

Component Based Communication – what we want

18

Web Services

Here is our architecture.

MySQL

JavaDatabase

Driver

JavaWeb Services

ApplicationTomcat

Web Server

Excel With Web Services

Reference Tool

We talk over the cloud using Web Services.

19

Web Services

Q: What is Web Services?

A: SOAP over a protocol such HTTP, JMS, SMTP, FTP, etc

HTTP is the most common protocol

20

Web Services

Web services allow us to pursue what I consider the Holy Grail ofthe software community for the last 20 years: reusable software.

We had a project estimated at about $800,000 using traditionaltechnology But by embracing some things weve done to exposeour legacy systems as Web services, they did the project for$30,000.

. . . John McKinley, CTO of Merrill Lynch

21

Web Services

Why is this technology so great?

I Uses open standards, e.g. HTTP, XML, SOAP

I It is platform independent

I Can be used to develop rich clients – in our case Excel

I Can be used by components people not necessary

22

Web Services

Some simple examples:

I Tracking packages

I Calculating shipping rates

I Shopping carts

I Automate ordering

I Credit card verification

I Optimization

23

Web Services

24

Web Services

25

Web Services

Here is the architecture once again.

MySQL

JavaDatabase

Driver

JavaWeb Services

ApplicationTomcat

Web Server

Excel With Web Services

Reference Tool

26

Web Services

The MySQL DBMS has three databases:

1. retailer

I a retailer table (universal relation)

I a sku table

I an orders table

I a supplier table

2. doopwop

I a records table

3. portoptdata

I a returns table

27

Web Services

The Web Services Holy Trinity

I SOAP (Simple Object Access Protocol) – This is theprotocol used to specify the format of the communicationpackets between two components over a network.

I WSDL (Web Services Discovery Language) – This is theprotocol used to describe the API provided by the Web service.

I UDDI (Universal Description, Discovery, and Integration) – This is the protocol that specifies how to publish anddiscover Web Services over a network.

28

Web ServicesSOAP Packets

29

Web Services

WSDL – see

I http://74.94.100.129:8080/sql/services/Sql?wsdl

Exercise – type this URLs into your browser!

The server returns the WSDL – the API (Application ProgrammingInterface) for communicating with the server.

30

Web Services

And finally . . .

After all of this ...

What is Web Services?

Nothing more than making function/method calls over the Web!

31

Web Services in MATLAB

Create the Web Service: What do you need to do? We illustratewith retailerSQL.m.

Step 1: define the WSDL

wsdl = ’http://74.94.100.129:8080/sql/services/Sql?wsdl’

Step 2: create a class using the wsdl

createClassFromWsdl(wsdl)

Step 3: create an object in the class

obj = SqlService;

BIG IDEA: The necessary code generated by MATLAB!

32

Web Services in MATLAB

Use the Web Service: What do you need to do?

Step 1: Define the arguments for the method used

Step 1a: Define a string variable with the SQL.

sqlquery1 = ’SELECT * FROM retailer.retailer’

Step 1b: Define the database.

database = ’retailer’;

Step 2: Call the executeSelect method.

sqlresult = execSelect(obj, database, sqlquery1)

33

Web Services in MATLAB

Here is the XML result of query:

sqlquery4 = strcat(’SELECT orders.sku_num, supplier.city’, ...

’ FROM orders INNER JOIN supplier ON’, ...

’ orders.sup_id = supplier.sup_id’, ...

’ WHERE orders.sku_num=577’ )

<sqlResult><status>Success</status>

<record>

<sku_num>577</sku_num><city>Plano</city>

</record>

<record>

<sku_num>577</sku_num><city>Lajitas</city>

</record>

</sqlResult>

34

Web Services in MATLAB

Key Concept:

I we get data from the Web

I the data we get is of type string

I we must parse the string

I it is important that the data be in a format that is easilyparsed

The modern data format is XML format – the topic of the nextmodule

35

XML Background

XML – some hype!

Today we’re at another important inflection point, embracing XMLin everything we do. – Bill Gates

Microsoft .NET is Microsofts platform for XML Web Services –Microsoft Annual Report

Sun strongly endorses XML for data interchange and is engineeringXML support into all of its relevant products and platforms. – SunWeb Site

36

XML Background

Consider the following scenarios:

I Problem with recipe data

I Problem with online clothing retailer (Key concept separatetext from data)

I Problem with company using reorder Web service

I Problem with integrating different software systems

37

XML Background

There are four key problems illustrated:

I We want data in an open format that easily exchanged

I We want to give meaning to the data (e.g. is the number 778a price, an SKU, an order quantity?)

I All parties must agree to the meaning of the data

I We want to separate text from data

38

XML Background

Summary – a file may have some or all of the following:

I text or content

I formatting information

I data

XML is about the data!

39

XML Background

File Types:

I binary (for example a relational database file)

I text

I flat file (e.g. comma delimited)

I data + markup (XML)

40

XML Background

Example:

http://faculty.chicagobooth.edu/kipp.martin/root/xml/

dooWopQuery.xml

I The root node, or element, is the sqlResult

I The root node has children nodes status record

I Each record has six children, id, title, group, year,price and url.

I XML is case sensitive

41

XML Background

<sqlResult>

<status>Success</status>

<record>

<id>1</id>

<title>In the Still of the Night</title>

<group>The Five Satins</group>

<year>1956</year>

<price>17</price>

<url>www.youtube.com/watch?v=4k9Dkz0DHSI</url>

</record>

.

.

.

42

XML Backgroud

XML has a Tree Structure

Go back to databaseCase.m and list all the records in the doowop

database table records.

I What is the root node?

I What are the children of the root node?

I What is the parent of a record node?

I What are the children of a record node?

43

XML Background

There are alternative ways to represent the data, here we useattributes in addition to elements

<record id ="1" title ="In the Still of the night"

group = "The Five Satins" year = "1956"

price = "17"

url="www.youtube.com/watch?v=4k9Dkz0DHSI" />

Attributes or elements?

I A row in a table is a good candidate for an element

I A column in a table is a good candidate for an attribute

44

XML Background

An XML document must be well formed to be parsed withouterror.

I the tags are case sensitive - <PRICE> and < /price> wrong

I the tags must be nested properly

<parent>

<child1>

<child2>

</child1>

</child2>

</parent>

I opening and closing tags are required

45

XML Background

Even better are valid XML documents

They are validated using a schema

The W3C controls the XML Schema standard. It is supported byall of the major companies. Schemas have xsd file extensions

IMPORTANT: Understand the difference between

I well formed XML

I valid XML

46

XML Background

In terms of object oriented programming we have the followinganalogy:

Schema ↔ Class

XML file ↔ Object

47

XML Background

A Record Class

<xs:complexType name="record">

<xs:sequence>

<xs:element name="id" type="xs:positiveInteger"/>

<xs:element name="title" type="xs:string"/>

<xs:element name="group" type="xs:string"/>

<xs:element name="year" type="xs:positiveInteger"/>

<xs:element name="price" type="xs:double"/>

<xs:element name="url" type="xs:anyURI"/>

</xs:sequence>

</xs:complexType>

Note the data types.

48

XML Background

A Record Object

<record>

<id>1</id>

<title>In the Still of the Night</title>

<group>The Five Satins</group>

<year>1956</year>

<price>17</price>

<url>www.youtube.com/watch?v=4k9Dkz0DHSI</url>

</record>

49

XML Background

Think of validation as verifying that an object is a proper memberof the class.

The earlier you can find errors the better.

You should validate an XML file before sending them up the foodchain.

You can do the validation in VBA.

50

XML Background

Key Idea: a schema (or set of related schemas) defines a markuplanguage or vocabulary

I AnatML Anatomical Markup Language

I XBRL eXtensible Business Reporting Language

I FpML Financial products Markup Language

I WordProcessingML, SpreadsheetML, DataDiagramingML

51

XML Background

Key Idea: a schema (or set of related schemas) defines a markuplanguage or vocabulary

I RecipeML a format for recipes

I MathML a format for representing math

I RSS a format for representing news feeds (also used for Webblogs)

52

XML Background

One Last Note: Once again, XML is NOT meant to be read bypeople.

53

Web Services

MATLAB provides a function createClassFromWsdl which makesusing Web Services amazingly easy.

The function createClassFromWsdl reads the URL of WebService as the argument.

The function createClassFromWsdl generates an object in a classbased on the WSDL specified by URL.

This class has the methods specified in the WSDL.

See parseSqlXML.m which generates the class SqlService.

54

Web Services

Here is how to use createClassFromWsdl:

wsdl = ’http://74.94.100.129:8080/sql/services/Sql?wsdl’’

createClassFromWsdl(wsdl)

55

Web Services

When you use the function createClassFromWsdl it will createan m-file for each method specified by the WSDL.

See the directory that gets created: @SqlService.

This directory has an m-file execSelect.m that implements theselect method. This method takes three arguments:

I the object SqlService

I the name of the database

I the SQL statement

56

Web Services

The m-files this figure were generated automatically. They arebased on the WSDL at:

74.94.100.129:8080/sql/services/Sql?wsdl

57

Web Services

Here is the MATLAB code for getting an SQL result:

obj = SqlService

sqlquery = ’SELECT * FROM doowop.records’

sqlresult = execSelect(obj, ’doowop’, sqlquery)

Here is the VBA code for getting the same SQL result:

Dim ws As clsws_SqlService

Set ws = New clsws_SqlService

Dim sqlquery As String

Dim xmlString As String

sqlquery = "SELECT * FROM doowop.records "

xmlString = ws.wsm_execSelect("doowop", sqlquery)

58

XML and DOM

MATLAB has features almost identical to VBA for reading XMLdocuments.

The key function is xmlRead. Assume that the file, test.xml,contains the data we wish to read. The following

xDoc = xmlread(’test.xml’);

is the equivalent of our VBA

Dim dooWopDom As DOMDocument60

Set dooWopDom = New DOMDocument60

dooWopDom.LoadXML (xmlString)

Both of these do exactly the same thing: they get the XML datainto an in-memory object that duplicates the structure of the XML– i.e. the parent children relationships.

59

XML and DOM

Now let’s do something with the XML. Refer to the fileparseSqlXML.m.

recordNodes = xDoc.getElementsByTagName(’record’);

n = recordNodes.getLength;

What does n equal? What does the code below do?

for k = 0:n-1

thisRecordNode = recordNodes.item(k);

childNode = thisRecordNode.item(0);

I why am I looping from 0 to n − 1?

I notice that, unlike in VBA, I do not declare thisRecordNode

and childNode to be an IXMLDOMNode.

60

XML and DOM

What about the following?

for k = 0:n-1

thisRecordNode = recordNodes.item(k);

childNode = thisRecordNode.item(0);

childNodeType = childNode.getNodeType

childNodeTagName = char(childNode.getTagName)

firstChildNodeType = childNode.getFirstChild.getNodeType

var1 = childNode.getFirstChild.getData

var2 = char(childNode.getFirstChild.getData)

var3 = str2num( char(childNode.getFirstChild.getData))

61

XML and DOM

Important to understand the data types of

var1 is Java string data

var1 = childNode.getFirstChild.getData

var2 MATLAB character data

var2 = char(childNode.getFirstChild.getData)

var3 finally convert to a number

var3 = str2num( char(childNode.getFirstChild.getData))

62

XML and DOM

Take the record:

<record>

<id>1</id>

<title>In the Still of the Night</title>

<group>The Five Satins</group>

<year>1956</year>

<price>17</price>

<url>www.youtube.com/watch?v=4k9Dkz0DHSI</url>

</record>

I Change the price from 17 to 27

I Add a <leadSinger> element and make Fred Paris the leadsinger.

63

XML and DOM

We want to end up with:

<record>

<id>1</id>

<title>In the Still of the Night</title>

<group>The Five Satins</group>

<year>1956</year>

<price>27</price>

<url>www.youtube.com/watch?v=4k9Dkz0DHSI</url>

<leadSinger>Fred Paris</leadSinger>

</record>

64

XML and DOM

Here is the additional code:

A(k + 1) = ...

str2num( char(childNode.getFirstChild.getData));

if A(k + 1) == 1; %we have In the Still of the Night

%change price of record 1

thisRecordNode.item(4).getFirstChild.setTextContent(’27’);

%add a lead singer

nameNode = xDoc.createElement(’leadSinger’);

nameNode.setTextContent(’Fred Paris’);

recordNodes.item(k).appendChild( nameNode);

end

65

XML and DOM

Class Exercise 1: write a MATLAB function namedfindGoogleDistance that takes two arguments. The firstargument is the location of an origin city and the second argumentis the location of the destination city. The function should returnthe Google distance.

I the function use user the MATLAB method urlread().

I the URL for urlread() should be appropriate address forGoogle Map

I parse the resulting XML using xmlread and get the distance

I return the distance

I the locations should be strings that look something like

’Chicago’

66

XML and DOM

Class Exercise 2: instead of writing the lead singer element at theend of the list, put the <leadSinger> after the element <year>.

Hints:

I http://docs.oracle.com/javase/6/docs/api/org/w3c/

dom/package-summary.html

I http://docs.oracle.com/javase/6/docs/api/org/w3c/

dom/Node.html

67

XML and DOM

Summary items:

I The DOM uses 0 based counting

I Use xmlwrite( xDoc) to write the result

I Use the setTextContent to change the content of an element

I Use xDoc.createElement to create a new element

I Use the appendChild method to add a newly created element

I Comment out the call createClassFromWsdl(wsdl) once youhave created the SqlService class

68