Software Factories : Assembling Applications with Patterns, Models, Frameworks and Tools Steve Cook Architect Visual Studio

Software Factories : Assembling Applications with Patterns, Models, Frameworks and Tools

Software Factories : Assembling Applications with Patterns, Models, Frameworks and Tools

Steve CookSteve CookArchitectArchitectVisual StudioVisual Studio

AgendaAgenda

Software IndustrializationSoftware IndustrializationUsing DSLs to Reduce ComplexityUsing DSLs to Reduce ComplexityScenario 1 - Web Services & Design Scenario 1 - Web Services & Design for Operationsfor OperationsScenario 2 – Visualizing CodeScenario 2 – Visualizing CodeScenario 3 – Implementing Business Scenario 3 – Implementing Business ProcessesProcessesSoftware FactoriesSoftware Factories

The Software Crisis (ca. 2004)The Software Crisis (ca. 2004)Development lags platform technologyDevelopment lags platform technology

Globally distributed business processes constructed line by Globally distributed business processes constructed line by lineline

$250B/yr in US (average $430K to $2.3M per $250B/yr in US (average $430K to $2.3M per project)project)

16% on time and budget but deliver less than planned (avg 16% on time and budget but deliver less than planned (avg 42%)42%)

53% overrun (avg 189%)53% overrun (avg 189%)

31% are canceled, losing $140B/yr31% are canceled, losing $140B/yr

Global demand will vastly exceed capacity this Global demand will vastly exceed capacity this decadedecade

Apprenticeship will not scale to level of demandApprenticeship will not scale to level of demand

Other industries industrialized to solve similar Other industries industrialized to solve similar problemsproblems

IndustrializationIndustrializationConfigure, adapt and assemble components to produce variantsConfigure, adapt and assemble components to produce variants

Standardize, integrate and automate production processesStandardize, integrate and automate production processes

Develop and configure extensible tools for rote or menial tasksDevelop and configure extensible tools for rote or menial tasks

MarketsStandardsProcessesContinuous improvement Simple &

manageable

“small-scale”

interfaces

Localization of

complexity

AgendaAgenda


Raising the Level of AbstractionRaising the Level of Abstraction

ExecutablesExecutables



Incr

easin

g Ab

stra

ctio

nIn

crea

sing

Abst

ract

ion

GapGap

RequirementsRequirements


LanguageLanguage

LanguageLanguage

Incr

easin

g Ab

stra

ctio

nIn

crea

sing

Abst

ract

ion

GapGap



LanguageLanguage

LanguageLanguage

Incr

easin

g Ab

stra

ctio

nIn

crea

sing

Abst

ract

ion

GapGap


The history of programming is an exercise in hierarchical

abstraction. In each generation, language designers produce

constructs for lessons learned in the previous generation, and

then architects use them to build more complex and

powerful abstractions. (Smith and Stotts)

C(x) h C(x) t 2m xih = –

Modelling and ProgrammingModelling and Programming

Textual Pictorial

Declarative

Procedural

class Magnox : NuclearPowerStation {

public void dropRods ()

jo biscuits 24 green

pat cakes 32 pink

Employeename

addresspromote

Jobdescription

payassign

0..*

a>b && c==d

call

CallRecord

call length

base rate: £/s

store

<CallRecord>

<caller><number>07713248</number>

Domain Specific Languages (DSLs) make problems “Small Scale”Domain Specific Languages (DSLs) make problems “Small Scale”

Small scale =Small scale =Not much design beyond existing platformNot much design beyond existing platform

easy to understandeasy to understand

agileagile

small team small team

early prototypesearly prototypes

easy maintenanceeasy maintenance

changes inexpensivechanges inexpensive

Finding prime numbers is Small Scale …Finding prime numbers is Small Scale …

If we have a computer that can:If we have a computer that can:do multiplication and divisiondo multiplication and division

do things repeatedlydo things repeatedly

_*__*__/__/_

_%__%_while (…) {…}while (…) {…}

C, Pascal, Fortran,…

Von Neumann computer with conventional CPU

language:platform:

Finding parts lists is Small Scale …Finding parts lists is Small Scale …

If we have a computer that can:If we have a computer that can:keep large lists of parts & scan them keep large lists of parts & scan them quicklyquickly

combine and filter the lists efficientlycombine and filter the lists efficiently

CREATE …CREATE …JOIN …… WHERE …

JOIN …… WHERE … SELECT …SELECT …

Relational Database SQL

platform: language:

Developing a GUI is Small Scale …Developing a GUI is Small Scale …

If we have a computer that can:If we have a computer that can:display combinations of windows and widgetsdisplay combinations of windows and widgets

show controls and interact with user in themshow controls and interact with user in them

Visual GUI builderSystem.Windows.Formsplatform: language:

if we have a if we have a languagelanguage of phone of phone billingbilling

and a and a platformplatform to run it on to run it on

Phone bill systems are Small Scale …Phone bill systems are Small Scale …

Phone Billing Engine

call

calendarmonth

CallRecord

call length

base rate: £/s

call length

friend discount rate: £/sstore

friends calls

other calls

-

bill

billing period

store

Telephone Billing Scheme

Anatomy of a DSLAnatomy of a DSL

Notation (concrete syntax)Notation (concrete syntax)May reuse predefined patterns eg box-line, channels, May reuse predefined patterns eg box-line, channels, grids, maps, layoutsgrids, maps, layouts

Concepts (abstract syntax)Concepts (abstract syntax)May reuse predefined patterns eg states, flows, May reuse predefined patterns eg states, flows, structuresstructures

Well-formedness rules (grammar rules)Well-formedness rules (grammar rules)Serialization format(s)Serialization format(s)

Typically XMLTypically XML

Interactive behaviourInteractive behaviourDrag/drop, copy/paste, select/edit, etcDrag/drop, copy/paste, select/edit, etc

Mappings and TransformationsMappings and Transformations

DSL & Framework PatternDSL & Framework Pattern

CustomCustomPartialPartial

ClassesClassesFrameworkFramework

PartialPartialClassesClasses

FrameworkFrameworkClassesClasses

FrameworkFrameworkConfig FilesConfig Files

Other Other Framework Framework

ArtifactsArtifacts

usesuses

usesuses

completescompletes

FrameworkFrameworkToolsTools

DSLDSLEditorEditor

DSLDSLDefinitionDefinition

generatesgenerates

OtherOtherEditorsEditors

edit / buildedit / build

generatesgenerates

AgendaAgenda


.asmx files .asmx code behinds source code Statements and attributes

XML ConfigurationFiles and schemas(e.g. WSDL files)

Project Files and templates

Deployment PoliciesAnd Packages

??

Scattered Concepts …Scattered Concepts …

IssuesIssuesConcept of Web service is “lost” among Concept of Web service is “lost” among many filesmany files

Developer must keep files in syncDeveloper must keep files in sync

Refactoring engines ignorant of Web Refactoring engines ignorant of Web service conceptsservice concepts

Can not process non-code files like schemas Can not process non-code files like schemas and config filesand config filesCannot undo refactoring changesCannot undo refactoring changes

Debugging, Intellisense, editors all Debugging, Intellisense, editors all ignorant of Web service conceptsignorant of Web service conceptsSource Control, Work Item, Tests all Source Control, Work Item, Tests all ignorant of Web Service conceptsignorant of Web Service concepts

A DSL In Use TodayA DSL In Use Today

.asmx files .asmx code behinds source code Statements and attributes

XML ConfigurationFiles and schemas(e.g. WSDL files)

Project Files and templates

Deployment PoliciesAnd Packages

…A Holistic View…A Holistic View

Web Service Connectivity DSL

Web Service Connectivity DSLWeb Service Connectivity DSL

Services, Messages, Services, Messages, Applications, EndpointsApplications, Endpoints


XML, Projects, XML, Projects, DBs, Classes, CodeDBs, Classes, Code


Physical Physical servers & servers & segmentssegments

DeploymentDeploymentUnitsUnits


Abstraction/Refinement

packaged into deployed on

IT Operations doesn’t IT Operations doesn’t understand application understand application requirementsrequirements

IT Operations doesn’t IT Operations doesn’t understand application understand application requirementsrequirements

DevelopmenDevelopmentt

IT OperationsIT Operations

Developers Developers don’t don’t understand understand operations operations policypolicy

Developers Developers don’t don’t understand understand operations operations policypolicy

Plan/DesignPlan/DesignPlan/DesignPlan/Design DevelopmentDevelopmentDevelopmentDevelopment TestingTestingTestingTesting DeploymentDeploymentDeploymentDeployment OperationsOperationsOperationsOperations

Design for OperationsDesign for Operations





Logical Logical Data CenterData Center







Constraints


AgendaAgenda


Code VisualizationCode Visualization












Constraints


AgendaAgenda


DSL-to-DSL MappingsDSL-to-DSL Mappings












Constraints


Business Processes Business Processes and Entitiesand Entities



Reconciliation

Invoice To PaymentInvoice To PaymentSeller OrgBuyer Org

Buyer’sFinancial ServicesProvider

Accounts PayableSupplier

B3.1 Receive, Process, &File Invoice

B3.2 Process 3-Way Match(PO, Receipt, Invoice)

Invoice (Original)/Debit Memo

Invoice

PO info

Procurement

Purchasing

Logistics and Distribution

WarehouseManagement

Receiving

B3.8 Remit Payment atSettlement Period

AP Clerk POV

B3.3 2-Way Match(Receipt, Invoice)

B3.15 Update VendorMaster (including set-up

customers as vendors forrefunds)

Vendor MasterData Store

Vendorinitiatedbillingprofile

changesB3.13 Receive Vendor

Profile Changes

B3.5 Reject/Dispute Invoice

Invoices (MatchException or

Out ofTolerance)

Out of ToleranceNotification

Remittance Advice & Payment (hardcopy)

B3.4 Invoice Approval

Invoice (&Image)

Data Store

Employee DataStoreApprover(s)

Project Accounting

Project Management

Business Intelligence

Business Intelligence

Manage Tax Compliance

Advisory

Human Resources

Time & ExpenseManagement

B3.7 Pending Invoices

Invoices(blocked -pendingreceipt)

Invoice (Non POtype (e.g. card,

direct order, etc))

Bank

B3.14 AuthorizeVendor Profile Changes

Invoices(cleared)

Invoices (w/inTolerance Match)

B3.6 Contact Supplier &Discuss Dispute/Rejection

Invoices(disputed, rejected)

B3.9 Respond to PaymentInquiry Remittance

Deposit

Vendorprofiledata

Acceptedprofile

changes

EmployeedataEmployee

data

Remittance Advice & Payment Release (electronic)

Vendor updateacknowledgement(approved, denied)

AgreementData Store

Agreement/Contract(negotiated terms, etc)

Vendorchanges

Approvals,rejections

Approval notifications

Requester orBuyer

(whoever owns txn)

Invoice (service based)

PO Change(out of tolerance

accepted)

Notification(rejected)

Invoices (MatchException or

Out ofTolerance)

Invoices (approved)

ReceiptNotice

Receipt Info

Invoice (w/inTolerance Match)

ReceiptedInvoices

ReceiptedInvoices

B3.10 Receive and reviewSummary Statement (card

based programs only)

Monthly Summary Invoice (card based programs only)Payment (to bank)

B3.11 Match SummaryStatement to card receipts/

invoicesstatement

Chargeback (Summary Invoice errors)

B3.12 Authorize/RejectPayment (by line item)

Card based receipts/invoices

statement

Employee

Billing Inquiry and ResponseNegotiation

Payment Inquiry/Invoice Status Response

Invoice Status Response

Financial Management

Electronic FundsProcessing

General Ledger

Capital AssetManagement

Treasury

Invoice,Payment JEs

Capital Goods Line Item Details

Cash DisbursementNeeds

Invoice Details

Remittance Advice & Payment (electronic)

Project Related Invoice Line Items

Tax Liability

Supplier Performance Data,Requester/Buyer

Performance Data,Financial Performance

Data,

Expense ReportExpense Reimbursement

Request

Payment (expense reimbursement)

Notification(accepted,rejected)

Refundchequeor EFT

Accounts Receivable

Refund payment

Credit memo

B3.16 Perform Prepayments

Prepayment

Credit Memo

Initial Prepayment

Dunning Letter

Example KPI’s:% Match (2-way)% Match (3-way)Invoice aging

FaxPhoneSnail MailE-MailEDI VANMarketPlace Exchange













Manage Tax Compliance


Buyer Org Perspective: Requisition to PaymentB3. Invoice to Payment (AP)

Acronyms: AP: Accounts Payable, JEs: Journal EntriesPO: Purchase Order, KPI: Key Performance Indicator

Preconditions: A purchase order has been created and sent to the Seller Organization.

A supplier invoice usually triggers the Invoice to Payment process. In process step B3.1, a supplier submits an invoice to the buyer organization, usually after they’ve picked, packed, and shipped product to the buyer (although pre-payments sometimes occur). The invoice is reviewed for accuracy and entered into a payables system for matching purposes. This can also trigger the flow of information to Capital Asset management tracking for depreciationpurposes, Project Accounting for project financials tracking, Tax for compliance reporting, and the General Ledger to account for liability owed. In process step B3.2, the buying organization’s Accounts Payable group/person then canperform a 3-way match between the original PO, the materials Receipt, and the Invoice. Process step B3.3 illustrates how some alternative purchasing models can be accommodated, such as direct orders, employee expense reportsubmission (which requires payment to the employee) or procurement card (like a credit card) based purchases, where a the requisition and PO is entirely bypassed. Process B3.4 allows for these alternative 3-way matching scenarios, as the invoice is approved and verified before payment is made to the vendor. In the event of match exception or out of tolerance scenarios (where amount invoiced is considerably different than expected), Accounts Payable notifiesthe requester of over tolerance (which could trigger a PO change) or rejects (disputes) the invoice (B3.5). In process step B3.6, Accounts Payable (or the Buyer) contacts the Supplier to resolve the rejected invoice. If matching andtolerance proceeds as planned, in process step B3.7 the invoice could be held in a pending state until a settlement period has been reached, optimizing working capital. Payment is then remitted to the supplier either electronically (viaa banking institution) or manually via a cheque-run (step B3.8). The invoice to payment process asserts a tight audit trail, which enables AP to know everything about the invoice and payment, supporting the ability to investigate andrespond to supplier payment issues (process B3.9). As mentioned above with the non-PO purchasing models, process steps B3.10 - B3.12 provide further details around procurement card purchasing verification and paymentprocessing since it evokes a different payment model - to the bank as opposed to the supplier (since payment has already been made). Finally, in steps B3.13 - B3.15 related vendor data maintenance processes are performed, whichplay an important role in ensuring correct and proper payment to suppliers, as well as avoiding late charges resulting from incorrect payments (locations, addresses, bank acct #s, etc).

Core Function/Module

LogicalData Store

Process

Inter-face

RelatedFunction/Module

Productoverlay

Data Flow

CollaborativeB2B Data Flow

B2B Data Flow

Key

start

AgendaAgenda


A Set of Interrelated ModelsA Set of Interrelated Models

Business Business CapabilitiesCapabilities


Manual Manual ProceduresProcedures


TechnologyTechnologyArchitectureArchitecture


ConstraintsConstraints

ReconciliationReconciliation







Physical servers Physical servers & segments& segments



Abstraction/Abstraction/RefinementRefinement


packaged intopackaged into deployed ondeployed on





… and related artifacts… and related artifacts















Physical servers Physical servers & segments& segments





packaged intopackaged into deployed ondeployed on





DSLsDSLsPatternsPatternsProcessesProcessesFrameworksFrameworksComponentsComponentsTransformsTransformsConstraintsConstraintsTest CasesTest Casesetc.etc.

…defines a Software Factory…defines a Software FactoryThis is called a software factory schemaThis is called a software factory schema

Like a recipe for a specific type of applicationLike a recipe for a specific type of application

A set of viewpoints related by mappings that support A set of viewpoints related by mappings that support transformation, validation, traceabilitytransformation, validation, traceability

Lists artifacts required to build application type and Lists artifacts required to build application type and explains how to combine themexplains how to combine them

A software factory template is contentA software factory template is contentConfigures a development environment for application Configures a development environment for application typetype

Projects, patterns, frameworks, guidanceProjects, patterns, frameworks, guidance

Configured development environment is software Configured development environment is software factoryfactory

Integrates tools, process and content for that type of Integrates tools, process and content for that type of applicationapplication

Domain specific editing, rendering, compilation, Domain specific editing, rendering, compilation, debugging, refactoringdebugging, refactoring

A Software Factory Defines A Domain-Specific MethodologyA Software Factory Defines A Domain-Specific Methodology

Domain specific guidance for application typeDomain specific guidance for application typeIntegrates tools, content and processIntegrates tools, content and process

Associates process fragments with viewpointsAssociates process fragments with viewpointsFragments are specific to their associated viewpointsFragments are specific to their associated viewpoints

Work flow is non-deterministic but constrainedWork flow is non-deterministic but constrained

What to model to build an applicationWhat to model to build an applicationHow to capture and analyze requirements, and map How to capture and analyze requirements, and map them to design and implementationthem to design and implementation

Identify types of models involved and show how they Identify types of models involved and show how they relate to each other and to other source artifactsrelate to each other and to other source artifacts

How models fit into the development processHow models fit into the development processHow to version them, debug them, generate tests from How to version them, debug them, generate tests from themthem

Building A FactoryBuilding A Factory

Build software factory schemaBuild software factory schemaDefine target application type and constituent viewpointsDefine target application type and constituent viewpoints

Viewpoints layer by major life cycle artifactViewpoints layer by major life cycle artifactRequirements, Architecture, Implementation, Deployment, Requirements, Architecture, Implementation, Deployment, Testing, Management, MaintenanceTesting, Management, Maintenance

Define life cycle process for target application Define life cycle process for target application typetype

Process fragments plus constraints on viewpoint Process fragments plus constraints on viewpoint transitionstransitions

Build software factory templateBuild software factory templateDelivered as nesting parameterized install packagesDelivered as nesting parameterized install packages

Refactor software factory as applications are builtRefactor software factory as applications are builtBased on new requirements and user feedbackBased on new requirements and user feedback

Using A FactoryUsing A FactoryDefine application to be developedDefine application to be developed

Only unique requirements – common requirements assumedOnly unique requirements – common requirements assumed

Specifying application customizes schema and Specifying application customizes schema and templatetemplate

Adds, removes or changes viewpointsAdds, removes or changes viewpoints

Customizes tool configurations and predefined contentCustomizes tool configurations and predefined content

Customizes process guidanceCustomizes process guidance

Use customized factory to develop application Use customized factory to develop application artifactsartifacts

Features outside factory scope custom developedFeatures outside factory scope custom developed

Refactor application as it evolvesRefactor application as it evolvesCapture changes in requirements in factory configurationCapture changes in requirements in factory configuration

Factory configuration defines delivered applicationFactory configuration defines delivered applicationSimplifies maintenance and enhancementSimplifies maintenance and enhancement

Makes impact of changes easier to understandMakes impact of changes easier to understand

Changes propagate through factoryChanges propagate through factory

The EndThe End


© 2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Documents

Software Factories : Assembling Applications with Patterns, Models, Frameworks and Tools Steve Cook Architect Visual Studio