Developing applications that stand the test of time

Cover Page

Uploaded June 27, 2011

Developing Applications that Stand the Test of Time

Author: Jeffrey G. Long ([email protected])

Date: April 15, 2003

Forum: Talk presented at the Collaborative Expedition workshop, sponsored by

the U.S. General Services Administration as part of the federal Chief Information

Officers’ Council.

Contents

Pages 1‐31: Slides (but no text) for presentation

License

This work is licensed under the Creative Commons Attribution‐NonCommercial

3.0 Unported License. To view a copy of this license, visit

http://creativecommons.org/licenses/by‐nc/3.0/ or send a letter to Creative

Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.

Leveraging Notational Systems: D l i li ti th t t d th t t f tiDeveloping applications that stand the test of time

Jeff LongApril 15, 2003

[email protected]

Proposed AgendaProposed Agenda1: Current software development1: Current software development

paradigm and consequences thereof (15 minutes)(15 minutes)

2: Proposed alternative paradigm and consequences thereof (15 minutes)consequences thereof (15 minutes)

3: How to implement rules as data (15 i t )minutes)

4/15/03 Copyright 2003 Jeff Long 2

Fundamental Hypothesis of Notational EngineeringMany problems in government science theMany problems in government, science, the

arts, and engineering exist solely because of the way we currently represent them. Thesethe way we currently represent them. These problems present an apparent “complexity barrier” and cannot be resolved with more computing power or more money. Their resolution requires a new abstraction which b th b i f t ti l l tibecomes the basis of a notational revolution and solves a whole class of previously-intractable problems


intractable problems.

A New Notational System Often Requires a Change of Paradigm

A way of looking at a subjectAn example, pattern, archetype, or y

model

A set of unconscious assumptions we have about a subjecthave about a subject


P t 1 Th C tPart 1: The Current Software DevelopmentSoftware Development Paradigm

Current ParadigmCurrent ParadigmComputer applications are defined inComputer applications are defined in

terms of algorithms and data

Algorithms are the rules which are used t i l t th d t d t d lto manipulate the data; data and rules are distinct

The model for this is the abacus


Algorithms are implemented as software Everything else, such as inventory y g , y

quantities or employee names, is implemented as datap

Thus most rules (e g work processesThus most rules (e.g. work processes, decision trees) are implemented as softwaresoftware


Immediate ConsequencesImmediate ConsequencesEvery application requires a lot ofEvery application requires a lot of

software, often >1 million linesNo one can really understand or manage– No one can really understand or manage this level of complexity

– There are numerous bugs in any softwareThere are numerous bugs in any software system

– Special tools and techniques are requiredSpecial tools and techniques are required to create and/or manage large bodies of code; these add to overall costs


There is a lot of “maintenance” of the software– The software has to be changed as bugs

are found and eliminatedTh ft h t b h d th– The software has to be changed as the rules change

– Rules change as users become more– Rules change as users become more sophisticated

– Rules change due to external pressures as g pwell (e.g. new languages or architectures; government requirements; competition)


The changes to software themselves introduce new bugsintroduce new bugs– Extensive testing is required in order to

have any comfort levelhave any comfort level– Extensive documentation is necessary, but

often skipped; maintenance programmers pp ; p gdon’t know why system was so designed


Additional AssumptionsAdditional AssumptionsUsers know what they want and canUsers know what they want and can

communicate that to programmersProgrammers who have experience in an– Programmers who have experience in an industry can understand user requirements

– User requirements can be documented in aUser requirements can be documented in a form that is less complex than the actual working systemg y

Software can be designed the same way as any other complex technology


way as any other complex technology

Results to DateResults to Date Increasing application backlog Increasing application backlog 2/6 development projects cancelled;

additional 3/6 considered failuresadditional 3/6 considered failuresTrue importance of very good designers

d i i land programmers increasingly recognized

Bugs have greater and greater consequences; viruses don’t help


High turnover of Chief Information Officers (average tenure: 18 months)( g )

Increasing use of packages– This typically forces changes in work– This typically forces changes in work

processes, or else requires expensive customization (>2x package price)( p g p )

Increasing use of offshore outsourcing (>35% by 2005?)(>35% by 2005?)


Lessons Learned?Lessons Learned? Subject experts cannot communicate all Subject experts cannot communicate all

requirements to programmers– their expertise took many years to acquire– their own understanding will evolve

Subject experts must see working prototypes, not paper representations

Subject experts must be able to directly and continuously update rules as needed

“Corporate” knowledge must be externalized


Part 2: The Ultra-Structure D l t P diDevelopment Paradigm

Where and How Should Rules be Represented in a System?Remove 99% of all rules from theRemove 99% of all rules from the

software Represent them in a standard If/ThenRepresent them in a standard If/Then

form (multiple Ifs, multiple Thens)Represent them as records of dataRepresent them as records of data

within a very small set of tables

Distinction between rules and data largely disappears!


largely disappears!

Immediate ConsequencesImmediate ConsequencesSoftware size is reduced by 2+ ordersSoftware size is reduced by 2 orders

of magnitude– simpler to create, manage, understand, p , g , ,

test, document, and teach– remaining software has no knowledge of

th ld it id b i t l l ithe world; it provides basic control logic that knows what tables to check in what order, how to resolve conflicts, etc.order, how to resolve conflicts, etc.

Software development team is very small and manageable


small and manageable

“Corporate” knowledge is externalized and is in a form anyone can see and understandy

Subject experts can enter, change, and otherwise manage rules (corporate k l d ) di l i h iknowledge) directly, without going to programmers for assistance

Knowledge is actionable not only by subject Knowledge is actionable not only by subject experts (e.g. as an encyclopedia) but also by the computer, for reasoning, decision support, etc.


Programmers do not need to know or understand all rules, just enough to determine the classes of rules and the proper animation procedures

Serious prototyping becomes feasible; communications with users improvesp

Testing & QA can be far more rigorousDocumentation can be more completeDocumentation can be more complete


Results to DateResults to Date First commercial system built with this

approach grew from $13M to $100M from 1986-2002, with $1K/month software expense; became largest independentexpense; became largest independent wholesaler in their industry partly due to low operating costsoperating costs

Other prototypes in language, biology, legal, games, and artificial life seem to confirmgames, and artificial life seem to confirm basic claims and expectations


Part 3: How to Implement R l D t (A P i )Rules as Data (A Primer)

Analysis of RulesAnalysis of Rules Statement of rules and device for executing Statement of rules and device for executing

them can be different; need not be software for both

Rules can be reformulated into a canonical form of “If a and b and c... then consider x and y and z”

One informal, “molecular” rule (e.g. “three ik d ’ ” i b b ll) bstrikes and you’re out” in baseball) may be

translated as many “atomic” rules


“If” values specify conditions under which each rule is examined; these arewhich each rule is examined; these are called “factors”

“Then consider” values specify addit Then consider values specify addit-ional criteria that must be considered before deciding what to do; these arebefore deciding what to do; these are called “considerations”


Factors become primary keys in third-normal-form RDBMS tablesnormal form RDBMS tables

Alternate keys can be specified if usefulT f th d f l bTens of thousands of rules can be grouped into 10-50 classes based on th i t d titheir syntax and semantics

These classes can be managed easily by the tools of a RDBMS


Control logic (“animation procedures”) reads relevant rules, including rules , gabout selecting rules, and carries out specified actions p

Once established, control logic should not need to change; all changes arenot need to change; all changes are made by subject experts who directly update the data (rules) of the systemupdate the data (rules) of the system


Separation of rules into categories defines tables in the RDBMS

There is no practical limit on number of rows in a table (i.e. > 1,000 rules is no problem)

Referential integrity and field edits ensure that rules maintain integrity

Queries, report writers and forms ease fthe review of rules by authorized users


Design can proceed by iterative prototype; small prototypes can easily evolve to necessary level of complexitynecessary level of complexity

Basic design process is to:– define what exists (existential rules)define what exists (existential rules)– define relations between these (network &

authorization rules)d fi ( t l & t t l l )– define processes (protocol & meta-protocol rules)

The application essentially becomes an Expert System using a RDBMSExpert System using a RDBMS


The Ruleform HypothesisThe Ruleform HypothesisComplex system structures are created by not-Complex system structures are created by not-

necessarily complex processes; and these processes are created by the animation of competency rules Competency rules can becompetency rules. Competency rules can be grouped into a small number of classes whose form is prescribed by "ruleforms". While the competency rules of a system change over timecompetency rules of a system change over time, the ruleforms remain constant. A well-designed collection of ruleforms can anticipate all logically possible competency rules that might apply to thepossible competency rules that might apply to the system, and constitutes the deep structure of the system.


The Theory Offers a Different Way toThe Theory Offers a Different Way to Look at Complex Systems and Processes

observables surface structuregenerates

rules

f l

middle structure

constrainsgroups of rules deep structure


The CoRE HypothesisThe CoRE HypothesisWe can create “Competency Rule Engines” or CoREsWe can create Competency Rule Engines , or CoREs,

consisting of <50 ruleforms, that are sufficient to represent all rules found among systems sharing broad family resemblances e g all corporationsbroad family resemblances, e.g. all corporations. Their definitive deep structure will be permanent, unchanging, and robust for all members of the family, whose differences in manifest structures andwhose differences in manifest structures and behaviors will be represented entirely as differences in competency rules. The animation procedures for each engine will be relatively simple compared toeach engine will be relatively simple compared to current applications, requiring less than 100,000 lines of code in a third generation language.


ReferencesReferences Long, J., and Denning, D., “Ultra-Structure: A design theory for

complex systems and processes.” In Communications of the ACM (January 1995)

Long, J., “A new notation for representing business and other rules.” In Long, J. (guest editor), Semiotica Special Issue on Notational Engineering, Volume 125-1/3 (1999)

Long, J., “How could the notation be the limitation?” In Long, J. (guest editor), Semiotica Special Issue on Notational Engineering, Volume 125-1/3 (1999)

Long, J., "Automated Identification of Sensitive Information in Documents Using Ultra-Structure". In Proceedings of the 20th Annual ASEM Conference, American Society for Engineering Management (October 1999)


Technology

Developing applications that stand the test of time