Putting Data into SOA

Putting Data into SOA:Data Virtualization, DataBuses, and Enterprise DataManagement

by Ken Orr, Fellow, Cutter Business Technology Council

One element in all of the discussions of service-oriented architecture

(SOA) implementation has been neglected — data. This Executive

Report highlights the major dimensions of what I consider SOA

architecture, focusing heavily on the critical importance of data

architecture in SOA governance, planning, and implementation.

Business Intelligence

Vol. 7, No. 11

http://www.cutter.com

About Cutter ConsortiumCutter Consortium is a unique IT advisory firm, comprising a group of more than 150 internationally recognized experts who have come together to offer content,consulting, and training to our clients. These experts are committed to delivering top-level, critical, and objective advice. They have done, and are doing, ground-breaking work in organizations worldwide, helping companies deal with issues inthe core areas of software development and agile project management, enterprisearchitecture, business technology trends and strategies, enterprise risk management,business intelligence, metrics, and sourcing.

Cutter delivers what no other IT research firm can: We give you Access to theExperts. You get practitioners’ points of view, derived from hands-on experience withthe same critical issues you are facing, not the perspective of a desk-bound analystwho can only make predictions and observations on what’s happening in themarketplace. With Cutter Consortium, you get the best practices and lessons learnedfrom the world’s leading experts, experts who are implementing these techniques at companies like yours right now.

Cutter’s clients are able to tap into its expertise in a variety of formats includingprint and online advisory services and journals, mentoring, workshops, training,and consulting. And by customizing our information products and training/consultingservices, you get the solutions you need, while staying within your budget.

Cutter Consortium’s philosophy is that there is no single right solution for allenterprises, or all departments within one enterprise, or even all projects within adepartment. Cutter believes that the complexity of the business technology issuesconfronting corporations today demands multiple detailed perspectives from which acompany can view its opportunities and risks in order to make the right strategic andtactical decisions. The simplistic pronouncements other analyst firms make do nottake into account the unique situation of each organization. This is another reason topresent the several sides to each issue: to enable clients to determine the course ofaction that best fits their unique situation.

For more information, contact Cutter Consortium at +1 781 648 8700 [email protected].

Cutter Business Technology Council

Access to the

Experts

Rob Austin Ron Blitstein Christine Davis Tom DeMarco Lynne Ellyn Jim Highsmith Tim Lister Lou Mazzucchelli Ken Orr Ed Yourdon


mailto:[email protected]


Service-oriented architecture(SOA) — a paradigm for“design, development, deploy-ment and management of aloosely coupled business appli-cation infrastructure.”1

Software module — “a small self-contained program that car-ries out a clearly defined task andis intended to operate within alarger program suite.”2

A CRITICAL POINT FOR SOA

Service-oriented architecture(SOA) is at that awkward, ado-lescent stage in the technology lifecycle that all important tech-nologies go through. When SOAwas only a concept, thoughtleaders could debate aboutwhat it meant, but no one really

questioned how great it was goingto be. Software was going to bebroken down into a set of clearlydefined, loosely coupled servicesthat would be commonly avail-able across the Internet via Web-based directories with servicesdefined in common, understand-able terms so that building Web-based applications would be assimple as plugging a lamp or cof-feemaker into a standard electri-cal plug or fitting a mouse intoa USB port (notice I resisted thetemptation to refer to LEGOs). Butnow, as people have begun todevelop larger and larger applica-tions using SOA, some of the origi-nal quibbles about definitions areassuming greater importance, andsome ideas that were initially sopromising seem to be in question.

As we know from the work ofresearchers like Everett Rogers3

and Clayton Christensen4 andpopularizers like Geoffrey Moore,5

most technology innovation tendsto play out in predictable states.In a 2002 Cutter Executive Report,6

I introduced my own technologymaturity curve that I developedin the 1980s: the guru gap curve(see Figure 1).

Like most major technologicalachievements, to paraphrasecomputer scientist Herb Grosch,much of what is good in SOA isnot new, and what’s new in SOAis not necessarily good. Compo-nent parts are not new, and mod-ular design is not new. Whatis new is the ability to developand deploy well-structured,

by Ken Orr, Fellow, Cutter Business Technology Council

Putting Data into SOA: Data Virtualization, Data Buses,and Enterprise Data ManagementBUSINESS INTELLIGENCEADVISORY SERVICEExecutive Report, Vol. 7, No. 11


Web-based applications that drawon enterprise data crying to getout. What is happening today isthat we in fact are rediscoveringthings that were perhaps betterunderstood in the 1970s and 1980sthan they are today. The aim ofthis Executive Report is to under-stand the vital importance thatdata plays in an SOA strategy. Todo that, we have to revisit thebasic assumptions of SOA andgive you some understanding ofwhy those conditions exist — inother words, rethink.

Today, SOA is at what I refer to inmy technology adoption modelas the “engineering phase.” LikeMoore’s chasm, my engineeringphase is a critical juncture thattechnologies have to get past if

they are to be successful. It is inthis period that individuals andorganizations start applying thetechnology to really large, com-plex problems and begin to morefully document their experiences.I believe the engineering phaseclosely describes the currentsituation with SOA: after years ofhype and hyperbole, ever largerSOA applications are now beingdeveloped and deployed in moreand more organizations to per-form more and more importantfunctions.

As in most things, there is goodnew and bad news. The goodnews is that individuals and organi-zations all over the world are learn-ing a great deal more about howto “do SOA” as well as gaining a

greater understanding of SOA’sstrengths and weaknesses. Thebad news is that SOA, like any seri-ous technology, has not been aseasy to implement as it was origi-nally made out to be and it has anumber of design challenges.

Finally, because SOA turns outto have many moving pieces,people-oriented elements likecollaboration, coordination, andgovernance are seen as increas-ingly important. For example, oneof the areas of contention in thenew world of SOA is who hasresponsibility for the data qualityand integrity of SOA applications:the developers or the databaseadministrators (DBAs)? Often,there is a conflict between thosewho look at all data as XML andthose that think of all data as rela-tional tables. Some call this an“impedance mismatch,” but,as we see, it is really more of asemantic confusion.

On the whole, I believe that SOAis a very good thing. I applaud thefact that SOA has brought modulardesign back into vogue, which isespecially important to the qualityof design of any large-scale appli-cation. I like the fact that SOA pro-motes standard functions andconnectivity.

But while it is a convergence ofa number of technology trends,

VOL. 7, NO. 11 www.cutter.com

22 BUSINESS INTELLIGENCE ADVISORY SERVICE

The Business Intelligence Advisory Service Executive Report is published by Cutter Consortium, 37 Broadway, Suite 1, Arlington, MA 02474-5552,USA. Client Services: Tel: +1 781 641 9876 or, within North America, +1 800 492 1650; Fax: +1 781 648 1950 or, within North America, +1 800 8881816; E-mail: [email protected]; Web site: www.cutter.com. Group Publisher: Chris Generali, E-mail: [email protected]. Managing Editor:Cindy Swain, E-mail: [email protected]. ISSN: 1540-7403. ©2007 by Cutter Consortium. All rights reserved. Unauthorized reproduction in any form,including photocopying, faxing, and image scanning, is against the law. Reprints make an excellent training tool. For information about reprintsand/or back issues, call +1 781 648 8700 or e-mail [email protected].

10 20

Time (years)

Productivity

Prototyping

Phase

Engineering

PhaseDeployment

Phase

Management

Interest Guru Predictions

Guru Gap

Actual

Productivity

2-4

Figure 1 — Orr technology adoption curve (the guru gap).





SOA is at base an architecture,not a product or even a set ofproducts. This has major implica-tions for those implementing SOA.Like data warehousing, whichwas also an architectural shiftwhen it first burst onto the scene,SOA will likely take the better partof another decade to succeed — ifit is going to succeed.

SOA continues a process that hasbeen going on for at least the last25 to 30 years in software develop-ment: the process that involvespulling apart the “independent”(orthogonal) dimensions of soft-ware. This process has alloweddesigners to be able to concen-trate on one aspect of a complexsystem at a time, while holding theother dimensions constant: a proc-ess borrowed from other mathe-matical, scientific, and engineeringactivities throughout history. Thepurpose of this Executive Report,then, is to highlight the majordimensions of what I considerSOA architecture, focusing heavilyon the critical importance of dataarchitecture in SOA governance,planning, and implementation.

We begin the report with a discus-sion of the separating of indepen-dent dimensions of software andwhy it matters to SOA developers.We then talk about the fundamen-tal nature of services and thedifferent kinds of services. Thereport then looks at softwarebuses before addressing the mostimportant dimension of SOA —that of enterprise data manage-ment and how it complements

SOA. The report then tackles theissue of reusability before con-cluding with a list of issues youneed to be aware of regardingthe road to SOA.

TEASING APART THE “BIG BALL OF MUD”

As introduced by Brian Foote andJoseph Yoder:

A BIG BALL OF MUD is hap-hazardly structured, sprawl-ing, sloppy, duct-tape andbailing wire, spaghetti codejungle. We’ve all seen them.These systems show unmis-takable signs of unregulatedgrowth, and repeated, expe-dient repair. Information isshared promiscuously amongdistant elements of the sys-tem, often to the point wherenearly all the important infor-mation becomes global orduplicated. The overall struc-ture of the system may neverhave been well defined. If itwas, it may have erodedbeyond recognition.7

In the beginning, individual pro-grams and systems resembled abig ball of mud (see Figure 2). All

the functionality was contained ina single framework: dimensionslike reporting, database, trans-action processing, presentation,workflow, security, and businessrules. Over time, these majordimensions have been teasedapart so that they can bedeveloped (and modified)independently.

The most important of these inde-pendent dimensions (in roughlythe historical sequence that theywere identified and isolated)include:

The reporting/BI dimension.In the earliest days of comput-ing, developers recognized thatall report programs followedpretty much the same hierar-chical pattern. Report gener-ators were the first of all theindependent dimensions tobe widely identified. The firstreport generators appeared inthe early 1960s and 1970s andled to fourth-generation lan-guages (4GLs) and a wholehost of BI tools.

©2007 CUTTER CONSORTIUM VOL. 7, NO. 11

EXECUTIVE REPORT 33

Traditional Application

Architecture

Component-Based

Application Architecture

Security

ComponentWorkflow

Component Transaction

Component

Presentation

Component

Reporting/BI

Component

Base

Application

Database

Component

Business

Rule

Component

Figure 2 — Breaking up the “big ball of mud.”

The database dimension. Therecognition of the database asan independent dimensionoccurred almost as early asthat of reporting. The first stageof database managementoccurred with the introductionof physical and then logical filesystems that made it possiblefor developers to access physi-cal data without knowing thephysical location or internalformat. The first real databasemanagement systems occurredin the early 1960s. Network,hierarchical, and inverted filedatabases were popular in the1960s, 1970s, and early 1980sbut were overtaken by rela-tional databases that first beganto emerge in the late 1970s andearly 1980s. In recent years,there has been increasingefforts to support object data-bases and XML structured data.

The transactional dimension.The need to separate out thetransaction component in largeonline applications was alsorecognized in the early 1960s,with the first and most populartransaction monitoring system,IBM’s CICS, being introduced inthe late 1960s. With each newgeneration of IT platforms(mainframes, client-server,Web-based, etc.), there hasbeen a recognition of the needfor some form of transactionprocessing capability.

The security dimension.Security became an issuewith the introduction of onlinesystems in the mid-1960s. The

introduction of LANs/WANs andthe Internet and the evolutionof viruses, worms, spam, andother forms of electronic attackhave greatly magnified theimportance of securitymanagement.

The presentation dimension.The user interface began withthe introduction of online sys-tems and remained relativelyconstant until the introductionin the late 1970s and early1980s of Windows-based userinterfaces. Xerox introducedone of the first true GUIs in theearly 1980s with its Star system,but it was Apple with the intro-duction of the Lisa and the Macthat greatly popularized thecapabilities of sophisticateduser interfaces. The introduc-tion of the Web, the browser,HTML, and multimedia has pro-duced the GUIs that we knowtoday. In addition, the constantintroduction of new wirelessplatforms (PDAs, cell phones,etc.) has led to the need forsophisticated GUIs targeted ata wide variety of devices.

The workflow dimension.Workflow was initially intro-duced in the late 1970s andearly 1980s largely to supportimage/document manage-ment. As organizations movedincreasingly to electronic formsof communication, there was aneed for some way to managethe routing of those electronicdocuments. By the mid-1990s,with the introduction of LANsand the Internet, worldwide

electronic routing became areality, and organizations look-ing to implement businessprocess management began tofocus on workflow for all sortsof processes.

The business rule dimension.Business rules have always rep-resented one of the knottiestproblems in application devel-opment. The early businessrule engines were introducedin the 1980s based on artificialintelligence technology. Inrecent years, there has been arediscovery of the importanceof business rules, and excitingresearch has been working tomake business rules as inde-pendent as, say, databasemanagement.

This breaking down of these indi-vidual dimensions of softwarehas taken decades and is stillgoing on. This process has madeit possible for developers to focustheir concerns without having toworry about everything at once.Unfortunately, since many of thedimensions have been identifiedat different times, a great manylegacy and other solutions eachhave their own variations of theother dimensions. For example,some database management sys-tems have their own business ruleengines, while various transactionsystems have their own process(workflow) engines. One of theproblems, then, is that tools devel-oped to solve one dimensionoften expand to encompass oneor more of the other dimensions




resulting in confusion, conflict,and added complexity.

The purpose of introducing thisclassification of software dimen-sions here is that SOA, if used cor-rectly, touches in different wayson all of these dimensions. In par-ticular, most SOA applicationshave to deal heavily with the fol-lowing dimensions: presentation,business process (workflow),business rules, and database(see Figure 3).

What Figure 3 shows is an ideal-ized environment (architecture) inwhich the business process is ini-tiated from the presentation layer(i.e., a process instance is cre-ated) and then the business proc-ess layer controls (orchestrates)the workflow between various“activities” done by specific“actors” (organizations, roles, orsystems). The business activitiesin these activities call (trigger)application components that, inturn, access and return data to theapplication components that thenperform business activities andinvoke business rules to operateon that data. Finally, the manipu-lated data is displayed by the pre-sentation layer.

The most important thing here iswhat is called the “separation ofconcerns.” The presentation layerknows only what it sends andreceives. In turn, each activitywithin the workflow only hasaccess to the small amount ofprocess instance data (e.g., cus-tomer ID, order ID, and so on, for

a sales order process) that itneeds to be concerned with. Theactivity knows what it receives,the data layer that it uses, and theoutput it sends back. The datalayer knows only what data isrequested or needs to be updated(what we refer to as data views).This environment provides a con-textual framework in which thevarious component parts arelargely isolated from the otherdimensions as much as possible.

Old hands at large-scale designand architecture may object thatmost of this is not new or relevantto SOA. While the first objectionis largely true (this overall archi-tecture is not entirely new), thesecond is not; this kind of archi-tecture is, in fact, quite relevantto SOA. Indeed, nothing in SOA

obviates the need for an overallframework where the nearly inde-pendent dimensions are carefullykept cleanly apart. The reasonwhy this is important to SOA isthat it is easy for people — in theirzeal to adopt a new paradigm —to forget where they are and whatthey’re trying to accomplish. Inthe end, we’ll use a variation ofFigure 3 as a model for under-standing how to develop variouskinds of services within an SOAframework.

The dimensions and frameworkshown here represent decades ofhard work, insight, and trial anderror. Each new generation ofsoftware architects and tool devel-opers seems to be prone to forgetwhy some of these dimensionsare important, believing somehow


EXECUTIVE REPORT 55

Presentation

(UI Services)

Business

Process

(Workflow

Services)

Applications

(Base/BR

Services)

Information

(Data

Services)

App A App B App C

Figure 3 — The relationship between UI (presentation), business process,base/business rules, and data management.

that because we have new tech-nologies we can skip over (oromit) one or more of thesedimensions. In my own work andin reviewing problem causes indeveloping very large systems, I’vefound that not having a soliddimensional framework makes itharder (sometimes impossible) tocome up with good, stable, long-term solutions. Eventually, thosedimensions overlooked comeback to bite developers and ulti-mately users.

In the next section, we talk aboutservices. While I know that read-ers have likely been exposed tothe basic definitions of servicesand SOA dozens, even hundreds,of times, we cover these defini-tions here to show that some ofthe problems that people are hav-ing with SOA are caused, in partat least, from starting with thewrong (or perhaps wrongheaded)definitions.

THE FUNDAMENTAL NATUREOF GOOD SERVICES

Well-Defined, Self-Contained,and Independent

A service is a function that iswell-defined, self-contained,and does not depend on thecontext or state of otherservices.

— Douglas K. Barry8

Most of the world’s big fights areover homonyms. I say “object,”and you say “object,” but wedon’t mean the same thing; I say“methodology,” and you say“methodology,” but we don’t

mean the same thing. We talk andtalk, but we don’t communicate.If the discussion is important,arguments ensue. If we’re lucky,we figure out we’re talking aboutdifferent things; if not, we keepquarrelling and maybe start a war.

And as certain concepts becomemore widely used, there areincreasingly heated discussions.Some of the homonym problemsare honest mistakes introduced bysimple misunderstandings, whileother problems are more mali-cious and are frequently caused,in the world of technology at anyrate, by an attempt to relabel anexisting product or approach soas to cash in on a new fad (e.g.,“new, improved with SOA!”).

Currently, “services” and “service-oriented architecture” are hotlycontested words and phrases. Notsurprisingly, there are a number ofdifferent meanings, so we’re goingto try to come up with some defi-nitions that, I hope, will help ushave some perspective about howto create an enterprise SOA envi-ronment. Let’s start by talkingabout services.

Services are an attempt to createa component-based, virtual, dis-tributed software developmentenvironment. One of the goals, itseems to me, is the desire to build(construct) software products forthe Web (remember “Web ser-vices” came before plain-old“services”) in a completelytransparent way.

As defined on SearchSOA.com:

Web services (sometimescalled application services)are services (usually includ-ing some combination ofprogramming and data, butpossibly including humanresources as well) that aremade available from abusiness’s Web server forWeb users or other Web-connected programs. Webservices range from suchmajor services as storagemanagement and customerrelationship management(CRM) down to much morelimited services such as thefurnishing of a stock quoteand the checking of bids foran auction item. The acceler-ating creation and availabilityof these services is a majorWeb trend.

Users can access some Webservices through a peer-to-peer arrangement ratherthan by going to a centralserver. Some services cancommunicate with otherservices, and this exchangeof procedures and data isgenerally enabled by a classof software known as mid-dleware. Services previouslypossible only with the olderstandardized serviceknown as Electronic DataInterchange (EDI) increas-ingly are likely to becomeWeb services. Besides thestandardization and wideavailability to users andbusinesses of the Internetitself, Web services are alsoincreasingly enabled by theuse of the Extensible MarkupLanguage (XML) as a meansof standardizing data formatsand exchanging data. XML isthe foundation for the WebServices DescriptionLanguage (WSDL).9




Notice that “Web services” aredefined here using the term “ser-vices” itself, that is, a case wherea phrase is defined in terms ofitself — a common but frequentlyirritating technique. Forget that fora moment and think of Web ser-vices in terms of functions, wherea function is defined as a trans-form that maps a set into a set ofinputs into outputs.10 A service inthis sense then is something that“does something.” In processterms, a value-added activity (akaservice) is one that adds value(i.e., does something useful).

Programming services or func-tions, for example, often returnsome output value(s) based oncertain input(s). A search func-tion, for example, returns a list ofanswers to a query or question(see Figure 4).

One of the common definitionsof services is that they are: (1)identifiable, (2) have well-definedinputs and outputs, and (3) arecomposable (i.e., multiple ser-vices can be strung together tomake a larger service).

The first notion that services areidentifiable means that they haveunique, recognizable names. Oneof the touted features of SOA isthat it will be based on a largenumber of uniquely identifiableservices and there will be onlinedirectories of common services(e.g., “process order,” “shipgoods”) that a software developercan search for, choose, andthen connect (compose) via

well-defined (standard) inputsand outputs.

Indeed, included in this idea isthat these directories would con-tain identified services, but theywould also refer to “serviceproviders” that would host(run/execute) the service onbehalf of the “service customer,”and all of the communicationbetween the various parties(service provider, directory, andservice consumer) would be car-ried out via standard message

protocols (see Figure 5). Whilethis framework represents philo-sophically the basis for a service-oriented future, the idea ofgeneralized SOA directory ser-vices seems to be less advancedthan that of internal directories,used by SOA development teamswithin individual organizations.Perhaps the presence of general-ized, Internet-wide SOA serviceswill eventually be achieved, butthe history of reuse is far morecomplex and difficult than most


EXECUTIVE REPORT 77

locations {from, to} Routing

Serviceroute segments

Figure 4 — A basic service.

Service

Consumer

Service

Provider

Service

Directory

Service Query

Service Description

Query Responses

Service Request

Service Response

Figure 5 — An SOA directory framework.

proponents admit. We will returnto this idea of reusability a bit laterin this report.

At any rate, SOA is currently pre-sented as a utopia of common,clearly identified, easy to find,easy to put together, stateless ser-vices. The technology involved ispretty well in place, but using thistechnology is much more difficultthan one might imagine, withmost of the difficulties stemmingfrom issues having to do withsemantics and understanding.Suppose we extend our routingservice in Figure 4. Figures 6a and6b, for example, show a slightlymore extended picture of howindependent services can becombined to create a larger ser-vice. Figure 6a shows the routingservice defined in terms of inputsand outputs in Figure 4, showingindividual input and output data,while Figure 6b shows a “map-ping service” that takes routing

segments and puts them out as aset of line segments overlaid ontoa map. Figure 7 shows a hierarchi-cal arrangement of these servicesnested within a “routing and map-ping service.”

This is how SOA is supposed towork, at least in theory. A com-plex function is broken down intoa series of simple functions or,conversely, created by combininga series of simpler functions intoa larger, more complex one.However, in this case, this powercomes with considerable develop-ment cost. Mapping software sys-tems such as MapQuest or GoogleMaps are the result of decades ofR&D involved in coming up withdigital mapping standards, digital(network) map databases, andsoftware to transverse these digi-tal maps to come up with routesand display the result. Hiddenfrom view in this service view ofthe world (or that of a series of

programming APIs for that matter)is the underlying databases andmetadata. To show this, I’ve cho-sen to use a diagramming stan-dard called IDEF011 in Figure 8. Inthis representation, the inputs stillcome in from the left and the out-puts still leave from the right, butwe’ve added a resource (in thiscase, database) coming in fromthe bottom and control (meta-data) coming in from the top.

Notice that the IDEF0 diagram hassome interesting similarities withthe architectural diagram in Figure3. Indeed, services, functions, andIDEF0 activities all share many ofthe same properties; they haveinputs, outputs, databases, andmetadata (control) information tofunction correctly. This is no acci-dent; the nature of IDEF0 processanalysis and that of SOA analysisand design turns out to be thesame — ensure a careful, correctmodular design.



Routing

Service

“From: 104 Woodlawn,

Topeka, KS”

“To: KCI Airport”

Mapping

Service

Figure 6a — Routing service with sample input and output.

Mapping

Service

Figure 6b — Mapping service with sample input and output.


Modularity, Coupling, Cohesion,Data Flow, and FunctionalDecomposition

Where the hot topics of softwaredevelopment today are “services,”“open,” and “agile,” in the 1970sand 1980s the hot topics were“structured,” “data structured,”“modular,” and “data flow.” Evenback then, systems were becom-ing larger and more complex, andit was becoming clear that soft-ware designs had to enforce somemore rational form of modulararchitecture.

Indeed, in order to build and testany very large system, it is neces-sary that the system be brokendown into pieces that can bedefined, implemented, and testedindependently of one another, thatis, modular. Therefore, all largesystems were modular, and sobeing modular was not enoughto guarantee success! What wasimportant, then, was which kindsof modular systems were theeasiest to develop, test, and mod-ify — in other words, what wasthe best kind of modularity? Theanswer about what were the keycharacteristics came from some

groundbreaking work by LarryConstantine and Cutter FellowEd Yourdon12 based on the analy-sis of the maintenance history oflarge systems. What they foundwas that the two key attributes ofgood modules were “coupling”and “cohesion.” Those modulesthat were the cheapest to main-tain were those that had lowcoupling and high cohesion. As itturned out, there came to be anappreciation about that time thatmodules with these attributeswere easier to develop and testas well.


EXECUTIVE REPORT 99

route segments route mapRouting

Service

Mapping

Service

Routing and Mapping Service

locations {from, to}

Figure 7 — Adding routing and mapping services to produce a routing and mapping service.

route segments

DigitalMap DB

metadata

locations {from, to} Routing

Service

Figure 8 — IDEF0 definition of routing service.

Low coupling means that mod-ules are not dependent on thefunctioning of any other modulesthat aren’t submodules of themodule itself. To repeat the quoteat the beginning of this section,a good service does not dependon the context or state of otherservices (modules). This meantthat pieces (modules) could beworked on independently, andteams could be assigned to workon these independent moduleswithout constantly meeting withevery other team. As a result, lowcoupling made both developmentand maintenance easier; pro-grammers or teams could focuson just their tasks at hand. Oneway to ensure that modules wereboth independent and had lowcoupling was to focus on modulesthat had only a single function,that is, they had high cohesion.

High cohesion means that a goodmodule does only one (or a smallnumber of) thing(s). Such mod-ules do not try to bundle a largenumber of functions within asingle module.13 Over time, highcohesion was shown to representa form of functional design thatfocused on defining modulesbased on generating outputs frominputs — in other words, some-thing called data flow design.

Data flow design is a method ofrepresenting software designs bybreaking down problems into setsof data flows by a process of func-tional decomposition (aka “top-down design”). Designers woulddevelop a picture of a system in

terms of the high-level inputs andoutputs from the system. Then thehighest-level design would be bro-ken down level by level with thedefinition of inputs from eachhigher level acting as a constrainton the next level of design. Ideally,this process would yield modulesand submodules that were consis-tent and highly cohesive. In prac-tice, the process was much morecomplex, but the concept of func-tional decomposition was easy tounderstand and teach and greatlyadvanced the movement toward amore functional view of softwaresystem design.

Interestingly, the ideas of modular-ity, coupling, cohesion, data flow,and functional decompositionhave come back into vogue as aresult of the increased interest inprocess (workflow) modeling andSOA. Services resemble well-formed modules more than theyresemble objects. In order to cre-ate good services that can bestrung together to create morepowerful services, there needsto be increasing thought given toinputs, outputs, database, andcontrols. But in addition to theidea of modularity, there are alsoa couple of other ideas that areextremely valuable in the searchfor an enterprise strategy for SOA:structured and data structureddesign, as we discuss next.

Structured and DataStructured Design

In the 1970s, software researcherscame up with some radicalbut simplifying ideas about the

organization of programs. Pro-grams that restricted themselvesto a form of highly organized struc-tures based on a very few funda-mental logical forms turned out tobe easier to develop and debugthan problems where the flow ofcontrol jumped willy-nilly fromone piece to another. This disci-pline became known as “struc-tured programming.” Softwareresearchers in Italy had discoveredthat all programs could be con-structed using only a handful oflogical constructs, but it was soft-ware researchers like EdsgerDijkstra, Harlan Mills, NiklausWirth, and others that made“structured programming” a majormovement. These basic structuresincluded only sequence, alterna-tion, and repetition initially, butsubsequent work discovered thatthere were two other structuresthat were also required for mostreal-time and complex modelingand design: concurrency andrecursion.

Sequence, of course, meant thatportions of a program could beorganized one piece after another.This was the most fundamentalkind of structure where Part A wasexecuted before Part B, and Part Bwas executed before Part C (i.e.,beginning, middle, and end).Alternation meant things couldtake only one leg or the other ofan if-then-else conditional con-struction.14 Repetition meant thatsubpieces of a program could berepeated, returning to the start ata starting point, again based onsome condition. Concurrency




meant that two (or more) piecescould be initiated at the sametime and processed in parallel,while recursion meant that pro-grams could be broken down intoa structure that actually invokedanother instance program itself asa subroutine (module) of itself.

Structured programs (and sys-tems) with modules that wereloosely coupled and highly cohe-sive turned out to be much easierto manage intellectually and as aresult easier to debug and main-tain. At a systems level, structuredbecame synonymous with dataflow design. As people began toconstrain program structures tothe basic set, multiple softwareresearchers noticed that in a vari-ety of circumstances the structureof the data and the structure ofthe program that operated lookedvery much alike. That identityspawned the idea of “data struc-tured design.”

Data flow design promoted think-ing about modular design in termsof the flow (transforms) of data.Data flow design allows the out-puts of activity A to be connectedto become the input to activityB, and the output of activity Bbecomes the input to activity C.Data flow design also supportedthe idea of functional decomposi-tion in which a complex functionwith well-defined inputs and out-puts could be broken down intoa set of connected, lower-levelfunctions, the first of which hadthe same input as the higher-levelfunction, and the final one having

the same output as the higher-level function. The route and map-ping function in Figure 7 could beseen as an example of functionaldecomposition.

Data flow design, it turns out, is avery old form of design. It is thetheme of one of the most impor-tant contributions that Unix madeto software development in itsearliest days — the concept of“pipelines” or “pipes.”15 A numberof common functions were imple-mented using standard implemen-tation of common text-processingfunctions (called “filters”), whichcould be connected in a variety ofways to produce different outputs,much as SOA gurus predict that itwill be possible shortly using com-mon services.

So how does this discussion ofcoupling, cohesion, data flow,and data structure help us under-stand SOA and, more importantly,understand the role that the enter-prise data layer plays in SOA?The answer is that, in a way, SOAdesign (at least the top-downdesign version of SOA design) canbe thought of as a subset of dataflow design. While SOA is oftenthought of as a natural extensionof OO and component-orienteddesign, it can just as easily beviewed as the rediscovery of mod-ular design since services andmodules are so outwardly similar.Many of the things that werelearned about data flow designare relevant once again. From thisviewpoint, it is interesting to viewmany of the current discussions

about how to do SOA design thathave already been discussed in asimilar but different context.

From my own practice, I havelearned in designing complex sys-tems that you need to come upwith a simple architecture thatisolates the various componentsin each independent (quasi)dimension — an architecture thatis both explainable and under-standable. SOA clearly involvesall the dimensions described inFigure 3, but we’re going to sim-plify our problem by focusing onjust three of those dimensions:process (workflow), services(application/business rules), anddata (information).

GET ON THE BUS(ES): THE PROCESS BUS, THE APPLICATION BUS, AND THE DATA BUS

In computer architecture, a buscan be defined as follows:

A bus (bidirectional universalswitch) is a subsystem thattransfers data or powerbetween computer compo-nents inside a computer orbetween computers, and abus typically is controlledby device driver software.Unlike a point-to-point con-nection, a bus can logicallyconnect several peripheralsover the same set of wires.Each bus defines its set ofconnectors to physically plugdevices, cards or cablestogether.16

The idea of a software bus thatis being talked about today isclearly inherited from computer


EXECUTIVE REPORT 1111

(electronic) hardware. Historically,buses have provided flexiblestructures for various electroniccomponents to communicate withone another. Such buses normallyrepresent standards agreed uponby the major players in the indus-try. The key components and theattendant different bus structuresare often referred to as “the hard-ware architecture.” Armed withstandard definitions that describehow one connects with a givenbus, manufacturers’ engineeringteams can design components orperipherals relatively independentof the organizations designing theother components. For decades,software engineers have striven toemulate the hardware folks.

Today, more than at any othertime, buses are the rage in the soft-ware world. While my colleagueAndy Maher says he dislikes theterm “bus” when associated withsoftware because it gives peoplethe illusion that there are real(physical) as opposed to virtualconnections involved, I, on theother hand, believe that buses,even if they are logical fictions, arevery useful fictions. Software busesmake it possible to lay out elegantframeworks from which it is possi-ble to build logical componentsand connect them in standardways, without having to worrymuch about the internal (physical)characteristics of the pieces. Youcan see from the earlier discussionabout teasing apart the variousquasi-independent softwaredimensions how useful softwarebuses can be in this endeavor. We

are going to use this approach topropose a solution for the idealarchitecture of the ultimate “soft-ware bus.”17

In order to think about a distrib-uted, Web-based, EnterpriseService Bus (ESB), we are goingto need at least three differentsub-buses by the time we’redone. The three most prominentare: (1) a process bus, (2) an application/business rule(service) bus, and, of course,a (3) data bus (see Figure 9).

The Enterprise Process(Workflow) Bus

Figure 9 describes a situation inwhich various activity services areexecuted by an overall businessprocess (workflow) managementscheme that allows for quasi-independent services to behooked together via a workflownetwork. One can think of a work-flow management system as aprocess bus that communicatesthe minimum information neces-sary between activities (outputs,inputs) and key process contextinformation. For example, in avehicle rental application the“rental agreement number” and“rental location” are likely to bepassed as inputs to each individ-ual service so that the basic proc-ess context can be inferred;information about decisions alsoneed to be passed from activitiesto decision points (e.g., “credit =‘approved’”).

With the increasing importanceof automated business process

management, there has beenconsiderable headway in develop-ing common business processlanguages for modeling (BPML)and execution (BPEL). One of thecontinuing headaches in processmanagement is the likelihood ofhaving more than one workflowprocess management engineinvolved. Increasingly, COTS pack-ages have adopted a standardworkflow engine to control theinner workings of their processes.Software vendor SAP, for example,which was one of the earliestadopters of a very strong processmanagement structure, has itsown Web-based process manage-ment scheme (NetWeaver), whileIBM and Microsoft each have theirown as well (i.e., WebSphere andBizTalk). Over time, standards likeBPML and BPEL will increasinglymake it possible for a smoothhandover across major (andminor) vendors independentof their specific process busimplementation.

A process bus, then, provides astandard harness, if you will, forconnecting process activitiestogether. Moreover, the processbus also provides the capabilityof maintaining what are calledthe “process instances” and, ofcourse, of managing the activity-to-activity flow. One of the enor-mously valuable secondarybenefits of using process buses isthat they collect a great deal ofimportant status and performancedata with little or no overt efforton the part of the people or sys-tems that execute the individual




activities. It is always possible toknow where “such and such areservation” is in the vehiclerental system by tracking theprocess instance of the one, forexample, with “rental agreementnumber = 123456.” (Figure 10shows a series of processinstances that have started atseparate times for a sales orderprocess; the activities (boxes) thatare darker represent the currentactive activity.)

Two terms that are commonlyassociated with businessprocesses are “orchestration” and“choreography,” as described byCarol McDonald:

Orchestration defines thecontrol and data flowbetween Web services toachieve a business process.Orchestration defines an“executable process” or therules for a business processflow defined in an XML doc-ument, which can be givento a business process engineto “orchestrate” the process,from the viewpoint of oneparticipant.

Choreography defines thesequence and dependenciesof interactions between mul-tiple roles to implement abusiness process composingmultiple Web services.Choreography describes thesequence of interactions forWeb service messages — it

defines the conditions underwhich a particular Webservice operation can beinvoked. WSDL describes thestatic interface, and choreog-raphy defines the “dynamic”behavior external interfacefrom a global view.

BPEL4WS primarily focuseson orchestration, while WSCIfocuses on choreography.With WSCI each participantin the message exchangedefines a WSCI interface.With BPEL4WS you describean executable process fromthe perspective of one of theparticipants.18

What the definitions above con-vey is that orchestration describes



Process ServicesProcess Services

Data Adapter Services

Data View Services

Data View Navigation Services

Core Enterprise Data Services

Application Services

Process Bus

Application

Bus

Data Bus

Enterprise

Process

(Workflow)

Metadata

Enterprise

Service/

Business Rule

Metadata

Data

Metadata

Figure 9 — The components of the service bus.

the high-level “logical” businessprocess, whereas choreographydescribes the detail physical,dynamic messaging that needs tobe sent, received, and monitoredto ensure that the physical imple-mentation mirrors what was actu-ally intended in the logical design.History has shown that softwaretool vendors are ultimately able tohide the physical details of imple-mentation from developers. Thefact that there are still discussionsabout choreography of businessprocesses indicates that there arestill issues with implementationof workflow management overthe Web. Designers should beextremely careful not to let cleverprogramming dictate the long-range design of key processes.

One of the key issues of enterprise-level workflow management isthe coordination of parallel, asyn-chronous activities across depart-mental, even enterprise, levels.Programming such activities hasalways been a tricky business. Thisis the domain of operating systemsand real-time control systems.However, with the worlds ofresearch, development, and expe-rience in such systems, comingfrom these domains, developerswill be less and less concernedwith the physical details.19

Defining the business process is acollaboration between businessusers and business process ana-lysts. Business process analysisand design provide the basis for“swimlane” models that, in turn,

provide the basis for the overallworkflow network and the identi-fication of the various activitieswithin the model. While activitiescan be completely or largely man-ual, most business process activi-ties in swimlane models todayinvolve either individuals actingin specific roles using some com-puter application (e.g., “checkingfor vehicle rates,” “confirming areservation”) or some completelyautomated activity (e.g., “calculatefinal bill”).

The Enterprise Application/Business Rule Bus (EAB)

The EAB is usually described as away for services to communicatewith one another over a distrib-uted environment using standardmessaging protocols. Within the



Accounting

Shop

Sales Manager

Order Entry

Credit Manager

Customer

entered order

No

approv ed order

shipping notice

billing notice

shipment

inv oice

pay ment

preapprov ed creditYesorder

Check Credi t

Allocate Goods

Ship Goods

Bill Customer

Accept and Pay

Process Payment

preapprovedOrder Entry

ACME Manufacturing -- Sales Order Process

1

Accounting

Shop

Sales Manager

Order Entry

Credit Manager

Customer

entered order

No

approv ed order

shipping notice

billing notice

shipment

inv oice

pay ment


Check Credi t

Allocate Goods

Ship Goods

Bill Customer

Accept and Pay

Process Payment



2

Accounting

Shop

Sales Manager

Order Entry

Credit Manager

Customer

entered order

No

approv ed order

shipping notice

billing notice

shipment

inv oice

pay ment


Check Credi t

Al locate Goods

Ship Goods

Bill Customer

Accept and Pay

Process Payment



3

Accounting

Shop

Sales Manager

Order Entry

Credit Manager

Customer

entered order

No

approv ed order

shipping notice

billing notice

shipment

inv oice

pay ment


Check Credi t

Allocate Goods

Ship Goods

Bill Customer

Accept and Pay

Process Payment



1

Accounting

Shop

Sales Manager

Order Entry

Credit Manager

Customer

entered order

No

approv ed order

shipping notice

billing notice

shipment

inv oice

pay ment


Check Credi t

Allocate Goods

Ship Goods

Bill Customer

Accept and Pay

Process Payment



1

Accounting

Shop

Sales Manager

Order Entry

Credit Manager

Customer

entered order

No

approv ed order

shipping notice

billing notice

shipment

inv oice

pay ment


Check Credi t

Allocate Goods

Ship Goods

Bill Customer

Accept and Pay

Process Payment



2

Accounting

Shop

Sales Manager

Order Entry

Credit Manager

Customer

entered order

No

approv ed order

shipping notice

billing notice

shipment

inv oice

pay ment


Check Credi t

Allocate Goods

Ship Goods

Bill Customer

Accept and Pay

Process Payment



2

Accounting

Shop

Sales Manager

Order Entry

Credit Manager

Customer

entered order

No

approv ed order

shipping notice

billing notice

shipment

inv oice

pay ment


Check Credi t

Al locate Goods

Ship Goods

Bill Customer

Accept and Pay

Process Payment



3

Accounting

Shop

Sales Manager

Order Entry

Credit Manager

Customer

entered order

No

approv ed order

shipping notice

billing notice

shipment

inv oice

pay ment


Check Credi t

Al locate Goods

Ship Goods

Bill Customer

Accept and Pay

Process Payment



3

Figure 10 — Business process instance.


context of our earlier discussionregarding the separation ofdimensions, we are going to focusthe domain of the EAB on just thebusiness application and businessrule portion of application devel-opment and what I call the dataview services, though the EABcould be, and often is, seen as thebus to end all buses. Whereas theprocess bus is concerned aboutprocess orchestration, the EAB isheavily involved in the registering,locating, and accessing of data,and with providing services withthe right data. The advantage ofdoing these functions over theInternet through common directo-ries is that, as in the old main-frame environment, softwarechanges don’t have to be down-loaded to millions (billions) ofclient systems in order for users tohave the latest updates. Changescan be made once and be avail-able immediately to everyone onthe Internet.

But as always, the devil is in thedetails. If you take the millions ofindividual pieces of millions ofapplications written in dozens oflanguages (both natural and com-puter), then the possibilities formisunderstanding are naturallyenormous. Therefore, the EABwithin our design structure isbroken into two functions: appli-cation services and data view ser-vices. Application services are thework to be done, while data viewservices are the data interface tothe internal data (enterprise data)that is needed to execute theapplication services. In my view,

this should be the minimum datanecessary to produce the rightinput from the input provided.The data view services shouldalso be responsible for initiatingthe updating of the enterprisedata as well.

The definition and design of appli-cation services and the data viewservices should be the responsi-bility of the application servicedesigners working with the busi-ness users and the business proc-ess analysts. There are a varietyof ways of specifying such dataviews, but I recommend the formshown in Figure 11.20 A developerdefining a data view service tosupport a business service doesn’tneed to know the underlyingsource or structure of the

enterprise data but should concen-trate on the data that is neededto make the service work anddescribe that data in terms thatare best understood to the user.

The Enterprise Data21 Bus

The third component of theEnterprise Service Bus, the enter-prise data bus (EDB), parallels theenterprise application bus andprovides the interface (linkage)between the individual data viewservices and the actual enterprisedata needed to retrieve andupdate that information. In a veryreal sense, the data bus is anarchitectural representation of anenterprise-level, distributed, het-erogeneous data managementsystem. The idea here is for theEDB to operate at the enterprise



Figure 11 — Data view definition, vehicle availability query.

level in much the same way thata DBMS (e.g., Oracle, DB2, SQLServer, MySQL) manages data fora set of applications within theenterprise. It is hoped that marketpressures will provide the impetusfor the major vendors to begin tobuild EDBs that will interface notonly with their own DBMSs butwith all matter of legacy andexternal data as well.

The EDB should be responsiblefor managing information forenterprise-level services to dataacross the enterprise, storedwherever it happens to be and inwhatever format it happens to bein. The EDB is a way of hiding thephysical location and physicalstructure of existing data, whichcurrently resides in some legacydata store, internal data store, orexternal data source.

The EDB allows developers and/orsoftware packages in the enter-prise application bus to ask forinformation structured preciselyfor their needs and lets the under-lying EDB functions access thatinformation and structure it appro-priately. The EDB should be a two-way street, that is, it should: (1)provide access to pull data fromthe ESB; and also (2) update datasupported by the EDB. If you lookat how many enterprise data ser-vices actually work, they do apretty good job today of accessingdata but not nearly so good a jobof supporting the update function.

The middle layer of the EDB con-tains a logical (and/or physical)

mapping of legacy data to com-mon business semantic entities.What is referred to here as thecore enterprise data is not unlikethe operational data stores thatoccur in data warehousingimplementations. In many SOAimplementations today, this is sup-ported almost entirely via meta-data and temporary copies ofdata. BEA, which has a very goodimplementation that it refers toas “data virtualization,” allowsfor the implementation of thetop two functions of the EDB.This is effective as long as the SOAis involved largely in businessintelligence functions (today, esti-mates are that 90% of SOA func-tions are BI-related). However, asSOA is extended to other moretransactional functions, therewill be an increasing need forthe middle layer of the EDB to beincreasingly real.

The lowest level of the EDB con-tains the “data adapters” thatexpose (attach) legacy data to theEDB. In enterprise architectureconsulting work, my colleaguesand I are constantly involved in try-ing to help clients figure out howto move from very old, legacyapplications whose data stillresides in obsolete database plat-forms. For the last two decades,organizations have attempted toreplace these core legacy systemswith some form of COTS applica-tion, with varying degrees of suc-cess. Not all of these attemptshave been successful and eventhe successful replacementstrategies have been traumatic.

Increasingly, organizations aretaking a longer view and thinkingabout using services to replacelegacy functions with Web-based,service-oriented applications whileusing a data strategy to slowlymigrate the legacy data to moremodern platforms.

WHAT LIES BEYOND:ENTERPRISE DATA SERVICES

In my discussions about andresearch into SOA, I have cometo believe that a number of reallyimportant trends are, with SOA,coming together. I believe thatthe real future of SOA is to becomethe architectural framework, notjust for new classes of “compositeapplications,” but for a broad classof enterprise applications. This isa trend that has been underwayfor some time. Over the pastfew decades, organizationshave moved from a set of quasi-independent, “siloed” applicationsto more enterprise-wide, businessprocess–driven applications andfrom application data managementto enterprise data management.

This architectural shift hasoccurred along a number ofstages. By the beginning of the1990s, for example, relationaldatabase researchers had devel-oped enterprise-wide approachesto distributed database manage-ment. DBMSs with the capabilityto support two-phase commitand replication were increasinglyavailable. At about the same time,the computing world shifted toclient-server, distributed appli-cations that would separate




presentation components fromapplication processing anddatabase manipulation.

And at the same time that organi-zations were beginning to unravelthe problems of client-server anddistributed computing, whichwere, by nature, multiplatform,enterprise-wide problems, thebusiness intelligence communitywas hard at work on the “enter-prise data warehousing problem”for integrating BI data across theentire enterprise.

Over this period, data warehous-ing became a major component ofa total enterprise data architecturedriven by a growing need to: (1)integrate data from multiple opera-tional systems for management

and marketing purposes and (2)to make it easier and faster toaccess that integrated data for BIpurposes (see Figure 12).22 Thisprocess was greatly accelerated bythe explosion of the Internet intocorporate computing.

Distributed computing, businessprocess management, and datawarehousing have all maturedgreatly since they were first pro-posed in the 1980s. In most cases,this has been a productive exer-cise. Organizations that haveembarked on and built robustdata warehousing architectures,for example, have been able togreatly enhance the ability ofmanagers, analysts, professionals,and operational personnel toaccess critical enterprise data.

The introduction of the Internethas only amplified the kinds andtypes of data that are now avail-able, and Web-based portals nowmake this enhanced data avail-able to larger and larger numbersof users (e.g., customers, businesspartners, media).

Over the next decade, organiza-tions, whether they use systemsrun on their own computersdeveloped by themselves or soft-ware vendors or they use hostedsystems operated in a software-as-a-service (SaaS) framework oncloud computing environments,will have to rethinking their datastrategies. No matter whose pro-grams an enterprise uses, thedata is its asset, perhaps its mostimportant asset. As more and



Figure 12 — Enterprise data flow architecture for data warehousing.

more organizations realize justhow valuable their data assets are,the more they are going to haveto manage that data, not in apiecemeal fashion, but from anenterprise-wide standpoint.

Enterprise data management isjust that — managing all of theenterprise’s data. Because of thecomplexity of large organizations’data, whatever scheme enter-prises come up with, they willnecessarily be distributed solu-tions. Developing an enterprisedata management (i.e., EDB) solu-tion within SOA represents a wayto recapture the high ground indata management. If SOA is anenterprise function, then enter-prise data management is as well.The EDB makes it possible tomigrate underlying applicationsand databases without anyoneknowing, which is really impor-tant, since in the next decadethere will be increasing attentionon rebuilding many enterprises’core business applications as allthe people who know those sys-tems as well as the computingindustry move to new platforms.

MISREADING THE HISTORY OFENGINEERING: REUSABILITY

The ultimate objective of SOA,or any other software approachfor that matter, is not reusability;the ultimate objective is better,cheaper, easier to maintaincomplex systems that supportbetter business processes and amuch higher degree of businessagility. Reusability may be oneof the ways to accomplish these

goals, but it is not the end in itself.Reusability ought to be the happyaccident of good design and theuse of component parts wherethose parts are available.

In manufacturing, reusability islargely the result of the high costof manufacturing the individualcomponent parts. In general, themore parts of any given kind youproduce, the lower the unit cost.Therefore, there is natural eco-nomic drive to use standardizedparts wherever possible. But, insoftware, the cost of manufactur-ing software parts, after the firstone is developed, is almost zero.Software costs all come fromdevelopment and testing, oftendevelopment of “almost” thesame thing over and over again.There is no manufacturing line,just an engineering department.

There are lessons to be learnedfrom the last 50-plus years of soft-ware development. Those soft-ware components that have beenmost reused have been those thatare well defined and well named.I suspect that the most successfulprogram of reuse in software his-tory was the Fortran scientific sub-routine library. The reason thatthis program was so successfulwas that it was built on hundreds(actually thousands) of years ofmathematical practice, and thatprovided a common set of func-tions that were precisely namedand precisely defined.

Mathematicians, engineers, andscientists throughout the worlduse the same symbols to mean

the same things and have beentaught to use the work of others.In addition, their work is func-tional, so that one can use succes-sive sets of operations to createnew operators. Moreover, mathe-matical functions are complexand require exhaustive testing. Allof these things lead scientists andengineers to seek out standardsoftware and not try to build abetter mousetrap.

Commercial software has beenfar more successful in definingcommon data than common func-tions or components. Today, wesee the successful “mashup” offunctions on the Internet, butwhen analyzed, it turns out thatunderlying these mashups arevery sophisticated, very complexdata structures that have beenworked out over very long periodsof time.

Any number of SOA programsthat I have seen spend enormouseffort trying to decide which“common services” to build, inthe hopes that those services willdramatically increase their devel-opment of complex applications.History has not been kind to thisapproach. This approach hasbeen tried without great successsince the mid-1980s using firstobjects and then components asthe rallying cry.

History has shown that the mostused functions are either very sim-ple ones that are used thousands(millions) of times per day or verylarge ones that do something verycomplex that is difficult to define




and debug. This report was gener-ated by an observation that thething that people seem to find themost useful from their SOA pro-gram is the common data services(e.g., “get customer info,” “getproduct status”).

Services, as has been pointed outhere, is a form of modular design,and the history of modular designis largely the one borrowed frombuilding architecture, “form everfollows function.” If we do a goodjob of modular design, and doa good job of keeping the proc-ess, the activities, and the datastraight, the reusable pieces willemerge naturally.

POTHOLES ON THEROAD TO NIRVANA

For those organizations that arejust starting down the road toSOA, this section addressessome of the things you need tobe aware of in the process.

Coarse-Grained and Fine-GrainedServices

You may have noticed that SOAexperts suggest that you focus on“coarse-grained” as opposed to“fine-grained” services. You mayalso see occasionally a suggestionthat you try to stay away fromapplications with very high trans-action volumes. All of this is aboutSOA’s current level of perfor-mance. Historically, there hasbeen a significant overheadinvolved in making the magic ofSOA happen. Therefore, thingsthat get executed a lot, that is,fine-grained services, are to be

avoided. The piece of the SOAiceberg that lies under the wateroften involves accessing data fromnon-SOA legacy systems and mak-ing it available to Web service-enabled applications.

How can you make sure thatyou’re not developing fine-grainedservices that will overwhelm yourperformance? Well, from a designpoint, start with top-down design.As you get further down the proc-ess chain, have developers anddatabase folks do estimates onthe number of accesses that criti-cal services may have during peakperiods of use. Get advice fromSOA experts on how to code thosecritical, fine-grained services.From a data access point, itmeans getting relatively largechunks of data when you do dataaccess. Don’t access individualattributes if you don’t have to.

The Cost of XML

SOA performance problems areexacerbated by the significantdata redundancy involved withXML messaging. While XML advo-cates expound on the fact thatXML messages are both computerand human readable, neithercomputers nor people much careabout this feature. Computersdon’t actually read the text part ofXML documents except to scanfor the “real data,” and peopleonly read them a relatively fewtimes, usually early in the develop-ment or when something goeswrong. On the other hand,millions of times per day XML

messages are handled by high-volume applications.

What all this means in executionis an enormous amount of redun-dancy in messages (often five to20 times the size of traditional flatencoded files), which requiresbigger, faster pipes. In addition,because of its hierarchical nature,XML messages are harder to vali-date as well, which adds addi-tional processing load.

Since so much of SOA is based onXML, what should you do if youhave lots of transactions? Well, theanswer is to look at which inter-faces are the most significant andto talk to experts about ways toremediate the problem. As thenumbers of messages per day orhour or minute mount, large orga-nizations are coming up withways to communicate load andprocessing time. In some cases,this means encoding and trans-mitting the XML tags only at thebeginning of a session, then usingthat information to compress themessages on sending and thenexpanding them on the receivingside. This adds parsing time atboth ends, but specialized com-munications devices will help inthis area in the future by buildingthe compression algorithms intothe microcode.

The Complexity of SOA

Another performance problem inthe world of Web services is thenumber of physical layers involvedin moving data up and down thephysical network that underlies



our Web-based applicationsand the resulting complexity. Inresearching this Executive Report,I found a very interesting reportfrom IBM on visualizing the exe-cution of Web Services,23 whichdescribes a tool that IBM hasdeveloped to help its SOA devel-opers track down performanceproblems.

Just looking at some of the graphsin that report shows the practicalproblems that SOA developers arelikely to have developing pro-grams that involve large numbersof applications and services run-ning on lots of different computerswith different loads, all connectedby many different telecommunica-tions pipes running at differentspeeds.

So what can you do if your perfor-mance is slow and you can’t fig-ure out the reason? Well, lookingat tools like IBM’s Web servicevisualization tool is one thing youcan do. Other vendors are sure tohave similar tools; find out whatthey are and get snapshots of howthe services are going to behaveon real applications.

The Overriding Importance of Data

I strongly believe that what I callthe enterprise data bus is criticalto getting the maximum value ofSOA over both the short and longrun. Think of the EDB as yourenterprise database manager.Make it as cleanly architected asyou can and constantly go backand refactor the overall business

semantic model whenever newdata structures are needed orwhen new attributes are added toexisting data views.

So what can you do to develop anenterprise data service mentality?Well, one thing is to look at toolsthat can bridge the gap. One toolthat I know called DeKlarit thatworks with Microsoft Visual Studiotakes sets of structures like theone in Figure 11, builds a com-mon data model, and automati-cally writes the navigation code toextract the data and presents it tothe programmer in the way that Idescribed earlier.

THE GHOSTS OF CHRISTMASPAST AND PRESENT: AD/CYCLE,THE IBM SAN FRANCISCOPROJECT, AND MICROSOFT’SPROJECT OSLO

When everybody knows thatsomething is so, it meansthat nobody knows nothin’.

— Andy Grove, cofounder ofIntel, in a 2005 interview

with Fortune

Like so many things, SOA is moreof a journey than a destination. Inthe late 1980s and early 1990s, Iworked with IBM on a very ambi-tious project called AD/Cycle.AD/Cycle was IBM’s answer toautomating the software process,what was called CASE at the time.IBM spent hundreds of millionson the project before it more orless slipped from the radar in themid-1990s.

Not long afterward, IBM startedanother project called the SanFrancisco Project. It was also amajor undertaking, this time todevelop a business technologystrategy to build a framework forbuilding business systems in Java.It too disappeared a few yearslater after absorbing more than$100 million. Neither AD/Cycle northe San Francisco Project weretotal losses for they producedtools that have found their wayinto the current Eclipse offerings;however, they did not produce theresults expected.

It turns out that coming up withan end-to-end solution — busi-ness strategy working code(and back) — is an enormousundertaking, too much for eventhe largest, most sophisticatedorganizations in the world.Microsoft has also been workingon its version of the end-to-endstructure, called Project Oslo. LikeAD/Cycle and the San FranciscoProject, there is a great deal ofemphasis on rapid, Web-based,model-driven development in thediscussions of Project Oslo, but ifexperience is any guide, Microsoftwill have more problems deliver-ing than one is led to believe. Thisis especially true when one con-siders the sheer complexity ofthe existing IT systems andinfrastructure.

It is reassuring that everyoneseems to be headed in the samedirection. However, AD/Cycle, theSan Francisco Project, and nowProject Oslo all promised great




things. All of these attempts hadhigh goals, had considerable buy-in from other software providersand customers, and were widelytouted in the technical press; andthey all consumed (or will con-sume) hundreds of millions ofdollars. Unfortunately, they didn’tconstitute the revolution every-body had been promised. Theydid, however, create a much morecomplex and sophisticated oper-ating environment. The worldclearly needs the Kool-Aid that thefolks are selling, but getting it andenjoying the benefits is not goingto be easy; the organizations thatsucceed will probably pick offpieces and get to the end pointfaster than those that take on amassive change all at one time.

In a way, selling SOA is a lot likeselling industrial grade trash com-pactors. While important, trashcompactors aren’t sexy; the sameis generally true about softwarearchitectures. New, glitzy systemsare sexy, new systems that bringin lots of new customers are sexy,but software architectures are not.Moving to a new systems architec-ture takes time and education.Moreover, there is the learningcurve. History has shown that thefirst use of a new technology typi-cally takes twice as long as thesecond. SOA is becoming moremature in those firms that havebeen using it for some time, butit is at ground zero in manyorganizations.

The good news is that thoseorganizations that do invest in

software architectures and stickwith the program usually reap therewards. The organizations thatinvested in data warehousing inthe 1990s, for example, are nowable to reap real rewards fromthose investments. The compa-nies that are able to take advan-tage of master data management,for example, are largely those thatmade those important invest-ments in the data warehousing.

My guess is that the organizationsthat take a long-range viewtoward SOA will be the ones thatbenefit the most from it. I knowof companies that have beenengaged on the path toward dis-tributed, flexible, model-drivenapplications for more than adecade now, going back to workon CORBA and DCOM and anynumber of other initiatives. Atbase, however, large-scale appli-cations still have the same com-ponents that we talked about atthe beginning of this report. Overtime, as we get smarter andsmarter, we will be able to teasethe parts apart and eat our reallybig elephant one bite at a time.

Before we conclude, I want tosay a word about SOA and gover-nance. One of the problems thatI’ve seen with some of my clientsis a conflict between the SOAdevelopers (mostly Java/XML or.Net folks) and DBAs. The mostserious issue I’ve seen had to dowith who had the right to updatewhat data. Now, I often end up onthe wrong side of debates withDBAs, but with respect to updating,

I believe that a representative ofthe owners of the data should beresponsible for defining the updat-ing rules of the source data. Now,in a great many cases, this hasn’tturned out to be a problem sincea huge percentage of the SOAdeployments are BI applications.

Finally, there is a strong dangerfrom an enterprise architecturestandpoint that SOA will turn intoa “systems veneer,” not unlike the“screen scraper” technology thatwas popular in the late 1980s andearly 1990s. The idea of compositeapplications makes it seem okayto build advanced systems basedon very old, very complex legacyapplications. In many cases, itseems to me, this is somewhatakin to building a very big deodor-izing plant over an open sewer —you can make things seem better,but you haven’t gotten at the rootproblem.

Old, complex legacy systemscannot just be hidden from viewbecause they are so bad and inter-faced with so many other systemsthat everyone is afraid of touchingthem. Using SOA as a means ofproviding slicker Internet inter-faces to integrated data achievesa fraction of the work that needsto be done.

On the other hand, people areusing SOA as a way to hide thetransition from old platforms anddatabases to new ones. I talked toa number of individuals workingon such problems. The dataadapters are built to “expose” key



data while hiding the structure ofthe underlying source; then thedatabase is reengineered piece bypiece, the data is converted, andthe data adapters are modified tocorrespond to the new databasedesign.

SOA represents a new way ofthinking about systems. Thereis a great deal to learn. Only byhaving a really solid architecturalframework and a plan to solveour enterprise long-term as wellas short-term problems will itachieve what really needs toachieved.

ENDNOTES

1FiereWorks. “SOA Definitions”(www.fiereworks.nl/soa-definitions.html).

2From the “IT Lecture Notes byMark Kelly: Official IT Glossary”(www.mckinnonsc.vic.edu.au/vceit).

3Rogers, Everett M. Diffusion ofInnovations. Free Press, 1995.

4Christensen, Clayton M. TheInnovator’s Dilemma. Collins, 2006.

5Moore, Geoffrey A. Crossing theChasm. Collins, 2006.

6Orr, Ken. “Managing TechnologyDecisionmaking.” CutterConsortium Business-IT StrategiesExecutive Report, Vol. 5, No. 9,September 2002.

7Foote, Brian, and Joseph Yoder.“Big Ball of Mud,” 1999(www.laputan.org/mud).

8Barry, Douglas K. Web Services andService-Oriented Architectures.Morgan Kaufmann, 2003.

9SearchSOA.com glossary (http://searchsoa.techtarget.com/gDefinition/0,294236,sid26_gci750567,00.html).

10A mathematical function is actuallydefined as a transform that mapsone or more inputs into a singleoutput.

11IDEF0 is an international standardthat was originally called SADTand developed by Doug Ross anda number of collaborators at acompany called SofTech in the1970s. It has proved to be a verypowerful way to describe complexfunctional process diagrams.

12Yourdon, Edward, and Larry L.Constantine. Structured Design.Prentice Hall, 1979.

13Notice that this is a rather differenttact than that of object-oriented(OO) design, where a large num-ber of behaviors (methods) arebundled with individual objects.

14A generalization of the if-then-elseconstruction was the CASE con-struction, a multiway branch.

15From the “Pipeline (Unix)” entryon Wikipedia: “In Unix-likecomputer operating systems, apipeline is … a set of processeschained by their standard streams,so that the output of each process… feeds directly as input … ofthe next one. Each connection isimplemented by an anonymouspipe. Filter programs are oftenused in this configuration. Theconcept was invented by Douglas

McIlroy for Unix shells and it wasnamed by analogy to a physicalpipeline” (http://en.wikipedia.org/wiki/Pipeline_(Unix)).

16See www.answers.com/topic/bus-computing.

17A note of caution: since softwarebuses are by nature virtual, it isimportant to examine proposedones for both logical consistencyand power. It is important not toconfuse international standardswith elegant or even workable busstructures. In the complex worldof Web-based software develop-ment, there are competing stan-dards, and just because one grouphas a solution that some of themajor players agree on doesn’tnecessarily mean that that stan-dard will win out in the market-place or even work for everyclass of problem.

18McDonald, Carol. “Orchestration,Choreography, Collaboration andJava Technology-based BusinessIntegration.” Carol McDonald’sBlog, 30 October 2003 (http://weblogs.java.net/blog/caroljmcdonald/archive/2003/10/orchestration_c.html).

19It is not only possible, it is criticalthat all but the most vital physicalissues be hidden from analysts,designers, and developers. Historyhas shown that every time we letpeople know too much about theinner workings of some softwaretool or its environment, theyinvariably take advantage of thepeculiarities of those inner work-ings so that when we decide tochange them, it is often a wrench-ing experience.




20The advantage of this form of datadefinition over XML is that thereare tools already available that willtake a large number of definitionslike the one in Figure 11 and auto-matically define a normalizeddatabase from them and automat-ically build the data navigation tomake the data view services oper-ate smoothly. Moreover, thesetools will redesign the databaseand/or build revised navigationwhen any of the data viewschange.

21Data in the context of the enter-prise data bus includes unstruc-tured data (e.g., e-mail, docu-ments, images, attachments,multimedia) as well as traditionalstructured data.

22A secondary, but important, objec-tive was to spread the computa-tional load across multiple plat-forms. In addition, instead ofexpecting operational systems tosupport increasingly large num-bers of reporting and BI demands,redundant “informational (datawarehouse/data marts) databas-es” were created on separateplatforms, freeing operationalsystems to focus on handlingtransactional data.

23De Pauw, W. et al. “Web ServicesNavigator: Visualizing theExecution of Web Services.”IBM Systems Journal, Vol. 44,No. 4, 2005.

ABOUT THE AUTHOR

Ken Orr is a Fellow of the CutterBusiness Technology Council anda Senior Consultant with CutterConsortium’s Agile Product &Project Management, BusinessIntelligence, Business-IT Strategies,and Enterprise Architecture prac-tices. He is also a regular speakerat Cutter Summits and symposia.Mr. Orr is a Principal Researcherwith the Ken Orr Institute, a busi-ness technology research organi-zation. Previously, he was anAffiliate Professor and Directorof the Center for the InnovativeApplication of Technology withthe School of Technology andInformation Management atWashington University. He is aninternationally recognized experton technology transfer, softwareengineering, information archi-tecture, and data warehousing.Mr. Orr has more than 30 years’experience in analysis, design,project management, technologyplanning, and managementconsulting. He is the author ofStructured Systems Development,Structured RequirementsDefinition, and The One MinuteMethodology. He can be reachedat [email protected].



Cons

ultin

g

BusinessIntelligence

Get Expert Advice on All the BusinessIntelligence Issues You FaceCutter Consortium offers advice and guidance from world-renowned consultants. The Consortiumfeatures a faculty whose expertise and credentials are unmatched by any other service provider.

Moreover, unlike many other consulting firms that use senior partners to sell a job but then assignjunior staff to actually perform the work at the customer’s site, Cutter has no junior staff anddeploys only its expert Senior Consultants, Fellows, and Technical Coaches on every assignment.Cutter’s expert practitioners have considerable management, technical, and domain-specificexperience assisting Fortune 500 and other organizations with everything from IT strategicplanning and organizational development to enterprise architecture, program management,data management strategies, benchmarking and measurement, and more.

In addition, Cutter does not rely on off-the-shelf solutions like so many consulting firms, butinstead customizes every solution to meet each client’s unique needs based on the clientorganization’s business drivers, culture, technology history, and budget.

The Consortium’s great strength is that it can draw on its more than 150 best-in-class consultantsto assemble the ideal team for your organization, tackling any challenge that might arise andoffering a complete solution from assessment through implementation.

Business Intelligence StrategyConsultingOur Senior Consultants consider yourbusiness needs as well as your infrastructurerequirements and help you prioritize thembased on criticality and potential return oninvestment (ROI).

We’ll help you develop a governance structurefor your business intelligence initiative thatincludes prioritization, funding of projects, andresolution of conflict — and we’ll guide youthrough the necessary steps to develop animplementation strategy.

Decision Support AssessmentConsultingThe implementation of a business intelligencesystem should be driven by a strong businessneed. There must be demonstrable ROI basedon a cost-benefit analysis. Before you embarkon such an initiative, Cutter Consortium canprovide you with a formal assessment of yourcurrent decision support environment basedon focus areas such as business value andstrategic direction, organizational reportingstructure, your current technical infrastructure,decision support applications, and futurepotential need.

Agile Data WarehousingCutter Consortium’s agile data warehousemethodology, pioneered by Senior ConsultantKen Collier and Cutter Practice Director JimHighsmith, recognizes that user requirementschange over time and thus makes use ofdevelopment techniques that are feature-driven, iterative, and collaborative. It will helpyour firm improve its business intelligencecapabilities using state-of-the art technologiesand practices. You’ll get actionable businessintelligence, providing insights into the busi-ness that can be leveraged into increasedrevenue and greater profit margins.

Agile data warehousing not only yieldsbenefits at twice the rate as compared with atraditional data warehousing plan, but yourteam will gain the infrastructure and skill setsneeded to repeat this success going forwardwithout the need for external resources.

A Cutter engagement can provide yourcompany with:

Design of data warehouse architecture,data model, and business logic forextractions and transformations

First-production version of a datawarehouse

BusinessIntelligence


Cutter Consortium | 37 Broadway, Suite 1, Arlington, MA 02474-5552, USA | Tel: +1 781 648 8700 | Fax: +1 781 648 1950 | www.cutter.com | [email protected]

ConsultingBusiness Intelligence

A foundation and core of agile datawarehousing expertise through effectiveknowledge transfer

A smoothly functioning businessintelligence team working at increasedlevels of productivity

Agile business intelligence training foryour business users

Data QualityOrganizations with top-quality data are betterpositioned to succeed. Exceptional data canenable a company to reduce costs, makesmarter decisions, and have more satisfiedcustomers.

The volume of data that flows through yourorganization every day is immense. Fromcustomer orders to changes in inventory levels,purchased demographic data, financial data,and more, Cutter Consortium can help youassess all your data quality needs, focus onyour most important data, and implement arigorous approach to data quality that returnsbenefits to the business:

More accurate and easier-to-understandfinancial reports

Data that better enables operationsdepartments to deliver the right goodsand services to the right customers,on time and at a lower cost

Improved market data resulting in a betterunderstanding of competitive position

More robust and accurate customerdata, improving the ability of the salesdepartment to maximize the lifetime valueof a customer

More trustworthy data entering theorganization from external suppliers

More pertinent, timely, and accurateinformation for decision makers

An opportunity to “build in” data quality

A corporate focus on the most importantdata

A data quality engagement may consist of:

A leadership roadmap to help leaders focusthe effort, build support, develop needed

policy and goals, measure progress, andmanage change

A data quality assessment comparing yourorganization’s data quality program againstbest practices, resulting in a series ofrecommendations

A study of your data supplier managementpractices and a suggested structure tosystematically improve quality and reducecosts associated with managing externalsuppliers of data; this effort can also bebeneficial when applied to your cross-organization information chain

A roadmap for building data quality intonew systems, such as CRM applications,to ensure these systems can really providethe business benefits they promise

A mentoring program, including educa-tion and real-world testing, to train anindividual(s) to be the data “maestro”or go-to person for your organization’sdata quality needs

Knowledge ManagementAssessments ConsultingThe breadth, depth, and volatility of theknowledge required to deliver world-classinformation systems is truly daunting. Learningmore efficient ways to find, access, and usethe expertise of people both inside and outsideone’s own organization can increase theeffectiveness of your application deliverymanagement.

Cutter Consortium Senior Consultants helpyou determine where and how to target yourknowledge management efforts and help yououtline a program to measure those efforts.

You’ll learn how to leverage the expertisethat already exists in your organization. We’llalso help you select the most appropriateknowledge management tools. Your resultingknowledge management activities willproduce real, traceable value to yourenterprise that blends both human andtechnological expertise.

CRM Readiness AssessmentCutter will assess your organization’s CRMenvironment and its supporting customer tech-nology to help it achieve its CRM goals. Thisassessment is designed to validate the CRMstrategy, boost the CRM technology initiative,and provide actionable recommendationsbased on a review of current CRM plans.

The benefits of the assessment include theidentification of potential CRM issues thatcould hinder ultimate use of customer tech-nology; the evaluation of current organizationstructure, culture, and CRM strategies; and theevaluation of the current technology environ-ment. The assessment will highlight strengths,risks, and opportunities for improvement, aswell as recommend next steps.

Cutter Consortium’s consultant compares yourcurrent environment against CRM strategy andtechnology best (and worst) practices across avariety of industries. The overall objectivessatisfied by the assessment are to examine thecultural, organizational, and strategic aspectsof the company’s business and informationtechnology environments as they pertain toCRM and any operational data store and datawarehouse efforts. Cutter will evaluate thefollowing key CRM areas:

Implementation of a coordinated, customer-focused business strategy

Creation of a CRM-friendly organizationstructure

Establishment of a CRM-savvy organizationculture

Utilization of a comprehensive definitionof “customer”

Implementation of an integrated customerinformation technology environment

In addition, Cutter will gauge how preparedyour enterprise is to efficiently utilize customer-oriented technology, including an operationaldata store, customer contact technology, anddata warehouse environment, and we willprovide specific, actionable optimization andrisk mitigation recommendations to improvethe potential for success in future or currentCRM initiatives.



Abou

t the

Pra

ctice Business Intelligence

PracticeThe strategies and technologies of business intelligence and knowledgemanagement are critical issues enterprises must embrace if they are to remaincompetitive in the e-business economy. It’s more important than ever to makethe right strategic decisions the first time.

Cutter Consortium’s Business Intelligence Practice helps companies take all theirenterprise data, augment it if appropriate, and turn it into a powerful strategicweapon that enables them to make better business decisions. The practice is uniquein that it provides clients with the full picture: technology discussions, productreviews, insight into organizational and cultural issues, and strategic advice acrossthe full spectrum of business intelligence. Clients get the background they need tomanage technical issues like data cleansing as well as management issues such ashow to encourage employees to participate in knowledge sharing and knowledgemanagement initiatives. From tactics that will help transform your company to aculture that accepts and embraces the value of information, to surveys of the toolsavailable to implement business intelligence initiatives, the Business IntelligencePractice helps clients leverage data into revenue-generating information.

Through Cutter’s subscription-based service and consulting, mentoring, and training,clients are ensured opinionated analyses of the latest data warehousing, datamining, knowledge management, CRM, and business intelligence strategies andproducts. You’ll discover the benefits of implementing these solutions, as wellas the pitfalls companies must consider when embracing these technologies.

Products and Services Available from the Business Intelligence Practice

• The Business Intelligence Advisory Service• Consulting• Inhouse Workshops• Mentoring• Research Reports

Other Cutter Consortium PracticesCutter Consortium aligns its products and services into the nine practice areasbelow. Each of these practices includes a subscription-based periodical service,plus consulting and training services.

• Agile Product & Project Management • Business Intelligence• Business-IT Strategies• Business Technology Trends & Impacts• Enterprise Architecture• Innovation & Enterprise Agility• IT Management• Measurement & Benchmarking Strategies• Enterprise Risk Management & Governance• Sourcing & Vendor Relationships

Senior ConsultantTeamThe Senior Consultants on Cutter’s BusinessIntelligence team are thought leaders in themany disciplines that make up businessintelligence. Like all Cutter ConsortiumSenior Consultants, each has gained a stellarreputation as a trailblazer in his or her field.They have written groundbreaking papers andbooks, developed methodologies that havebeen implemented by leading organizations,and continue to study the impact thatbusiness intelligence strategies and tactics arehaving on enterprises worldwide. The teamincludes:

• Verna Allee• Ken Collier• Lance Dublin• Clive Finkelstein• David Gleason• Curt Hall• David C. Hay• Vince Kellen• David Loshin• Larissa T. Moss• Ken Orr• Gabriele Piccoli• Thomas C. Redman• Ricardo Rendón• Michael Schmitz• Ed Schuster• Karl M. Wiig


Documents

Putting Data into SOA