View
330
Download
0
Category
Preview:
DESCRIPTION
April 15, 2003: “Leveraging Notational Systems: Developing applications that stand the test of time”. Presented at the Collaborative Expedition workshop, sponsored by the U.S. General Services Administration as part of the federal Chief Information Officers’ Council.
Citation preview
Cover Page
Uploaded June 27, 2011
Developing Applications that Stand the Test of Time
Author: Jeffrey G. Long (jefflong@aol.com)
Date: April 15, 2003
Forum: Talk presented at the Collaborative Expedition workshop, sponsored by
the U.S. General Services Administration as part of the federal Chief Information
Officers’ Council.
Contents
Pages 1‐31: Slides (but no text) for presentation
License
This work is licensed under the Creative Commons Attribution‐NonCommercial
3.0 Unported License. To view a copy of this license, visit
http://creativecommons.org/licenses/by‐nc/3.0/ or send a letter to Creative
Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.
Leveraging Notational Systems: D l i li ti th t t d th t t f tiDeveloping applications that stand the test of time
Jeff LongApril 15, 2003
jefflong@aol.com
Proposed AgendaProposed Agenda1: Current software development1: Current software development
paradigm and consequences thereof (15 minutes)(15 minutes)
2: Proposed alternative paradigm and consequences thereof (15 minutes)consequences thereof (15 minutes)
3: How to implement rules as data (15 i t )minutes)
4/15/03 Copyright 2003 Jeff Long 2
Fundamental Hypothesis of Notational EngineeringMany problems in government science theMany problems in government, science, the
arts, and engineering exist solely because of the way we currently represent them. Thesethe way we currently represent them. These problems present an apparent “complexity barrier” and cannot be resolved with more computing power or more money. Their resolution requires a new abstraction which b th b i f t ti l l tibecomes the basis of a notational revolution and solves a whole class of previously-intractable problems
4/15/03 Copyright 2003 Jeff Long 3
intractable problems.
A New Notational System Often Requires a Change of Paradigm
A way of looking at a subjectAn example, pattern, archetype, or y
model
A set of unconscious assumptions we have about a subjecthave about a subject
4/15/03 Copyright 2003 Jeff Long 4
P t 1 Th C tPart 1: The Current Software DevelopmentSoftware Development Paradigm
Current ParadigmCurrent ParadigmComputer applications are defined inComputer applications are defined in
terms of algorithms and data
Algorithms are the rules which are used t i l t th d t d t d lto manipulate the data; data and rules are distinct
The model for this is the abacus
4/15/03 Copyright 2003 Jeff Long 6
Algorithms are implemented as software Everything else, such as inventory y g , y
quantities or employee names, is implemented as datap
Thus most rules (e g work processesThus most rules (e.g. work processes, decision trees) are implemented as softwaresoftware
4/15/03 Copyright 2003 Jeff Long 7
Immediate ConsequencesImmediate ConsequencesEvery application requires a lot ofEvery application requires a lot of
software, often >1 million linesNo one can really understand or manage– No one can really understand or manage this level of complexity
– There are numerous bugs in any softwareThere are numerous bugs in any software system
– Special tools and techniques are requiredSpecial tools and techniques are required to create and/or manage large bodies of code; these add to overall costs
4/15/03 Copyright 2003 Jeff Long 8
There is a lot of “maintenance” of the software– The software has to be changed as bugs
are found and eliminatedTh ft h t b h d th– The software has to be changed as the rules change
– Rules change as users become more– Rules change as users become more sophisticated
– Rules change due to external pressures as g pwell (e.g. new languages or architectures; government requirements; competition)
4/15/03 Copyright 2003 Jeff Long 9
The changes to software themselves introduce new bugsintroduce new bugs– Extensive testing is required in order to
have any comfort levelhave any comfort level– Extensive documentation is necessary, but
often skipped; maintenance programmers pp ; p gdon’t know why system was so designed
4/15/03 Copyright 2003 Jeff Long 10
Additional AssumptionsAdditional AssumptionsUsers know what they want and canUsers know what they want and can
communicate that to programmersProgrammers who have experience in an– Programmers who have experience in an industry can understand user requirements
– User requirements can be documented in aUser requirements can be documented in a form that is less complex than the actual working systemg y
Software can be designed the same way as any other complex technology
4/15/03 Copyright 2003 Jeff Long 11
way as any other complex technology
Results to DateResults to Date Increasing application backlog Increasing application backlog 2/6 development projects cancelled;
additional 3/6 considered failuresadditional 3/6 considered failuresTrue importance of very good designers
d i i land programmers increasingly recognized
Bugs have greater and greater consequences; viruses don’t help
4/15/03 Copyright 2003 Jeff Long 12
High turnover of Chief Information Officers (average tenure: 18 months)( g )
Increasing use of packages– This typically forces changes in work– This typically forces changes in work
processes, or else requires expensive customization (>2x package price)( p g p )
Increasing use of offshore outsourcing (>35% by 2005?)(>35% by 2005?)
4/15/03 Copyright 2003 Jeff Long 13
Lessons Learned?Lessons Learned? Subject experts cannot communicate all Subject experts cannot communicate all
requirements to programmers– their expertise took many years to acquire– their own understanding will evolve
Subject experts must see working prototypes, not paper representations
Subject experts must be able to directly and continuously update rules as needed
“Corporate” knowledge must be externalized
4/15/03 Copyright 2003 Jeff Long 14
Part 2: The Ultra-Structure D l t P diDevelopment Paradigm
Where and How Should Rules be Represented in a System?Remove 99% of all rules from theRemove 99% of all rules from the
software Represent them in a standard If/ThenRepresent them in a standard If/Then
form (multiple Ifs, multiple Thens)Represent them as records of dataRepresent them as records of data
within a very small set of tables
Distinction between rules and data largely disappears!
4/15/03 Copyright 2003 Jeff Long 16
largely disappears!
Immediate ConsequencesImmediate ConsequencesSoftware size is reduced by 2+ ordersSoftware size is reduced by 2 orders
of magnitude– simpler to create, manage, understand, p , g , ,
test, document, and teach– remaining software has no knowledge of
th ld it id b i t l l ithe world; it provides basic control logic that knows what tables to check in what order, how to resolve conflicts, etc.order, how to resolve conflicts, etc.
Software development team is very small and manageable
4/15/03 Copyright 2003 Jeff Long 17
small and manageable
“Corporate” knowledge is externalized and is in a form anyone can see and understandy
Subject experts can enter, change, and otherwise manage rules (corporate k l d ) di l i h iknowledge) directly, without going to programmers for assistance
Knowledge is actionable not only by subject Knowledge is actionable not only by subject experts (e.g. as an encyclopedia) but also by the computer, for reasoning, decision support, etc.
4/15/03 Copyright 2003 Jeff Long 18
Programmers do not need to know or understand all rules, just enough to determine the classes of rules and the proper animation procedures
Serious prototyping becomes feasible; communications with users improvesp
Testing & QA can be far more rigorousDocumentation can be more completeDocumentation can be more complete
4/15/03 Copyright 2003 Jeff Long 19
Results to DateResults to Date First commercial system built with this
approach grew from $13M to $100M from 1986-2002, with $1K/month software expense; became largest independentexpense; became largest independent wholesaler in their industry partly due to low operating costsoperating costs
Other prototypes in language, biology, legal, games, and artificial life seem to confirmgames, and artificial life seem to confirm basic claims and expectations
4/15/03 Copyright 2003 Jeff Long 20
Part 3: How to Implement R l D t (A P i )Rules as Data (A Primer)
Analysis of RulesAnalysis of Rules Statement of rules and device for executing Statement of rules and device for executing
them can be different; need not be software for both
Rules can be reformulated into a canonical form of “If a and b and c... then consider x and y and z”
One informal, “molecular” rule (e.g. “three ik d ’ ” i b b ll) bstrikes and you’re out” in baseball) may be
translated as many “atomic” rules
4/15/03 Copyright 2003 Jeff Long 22
“If” values specify conditions under which each rule is examined; these arewhich each rule is examined; these are called “factors”
“Then consider” values specify addit Then consider values specify addit-ional criteria that must be considered before deciding what to do; these arebefore deciding what to do; these are called “considerations”
4/15/03 Copyright 2003 Jeff Long 23
Factors become primary keys in third-normal-form RDBMS tablesnormal form RDBMS tables
Alternate keys can be specified if usefulT f th d f l bTens of thousands of rules can be grouped into 10-50 classes based on th i t d titheir syntax and semantics
These classes can be managed easily by the tools of a RDBMS
4/15/03 Copyright 2003 Jeff Long 24
Control logic (“animation procedures”) reads relevant rules, including rules , gabout selecting rules, and carries out specified actions p
Once established, control logic should not need to change; all changes arenot need to change; all changes are made by subject experts who directly update the data (rules) of the systemupdate the data (rules) of the system
4/15/03 Copyright 2003 Jeff Long 25
Separation of rules into categories defines tables in the RDBMS
There is no practical limit on number of rows in a table (i.e. > 1,000 rules is no problem)
Referential integrity and field edits ensure that rules maintain integrity
Queries, report writers and forms ease fthe review of rules by authorized users
4/15/03 Copyright 2003 Jeff Long 26
Design can proceed by iterative prototype; small prototypes can easily evolve to necessary level of complexitynecessary level of complexity
Basic design process is to:– define what exists (existential rules)define what exists (existential rules)– define relations between these (network &
authorization rules)d fi ( t l & t t l l )– define processes (protocol & meta-protocol rules)
The application essentially becomes an Expert System using a RDBMSExpert System using a RDBMS
4/15/03 Copyright 2003 Jeff Long 27
The Ruleform HypothesisThe Ruleform HypothesisComplex system structures are created by not-Complex system structures are created by not-
necessarily complex processes; and these processes are created by the animation of competency rules Competency rules can becompetency rules. Competency rules can be grouped into a small number of classes whose form is prescribed by "ruleforms". While the competency rules of a system change over timecompetency rules of a system change over time, the ruleforms remain constant. A well-designed collection of ruleforms can anticipate all logically possible competency rules that might apply to thepossible competency rules that might apply to the system, and constitutes the deep structure of the system.
4/15/03 Copyright 2003 Jeff Long 28
The Theory Offers a Different Way toThe Theory Offers a Different Way to Look at Complex Systems and Processes
observables surface structuregenerates
rules
f l
middle structure
constrainsgroups of rules deep structure
4/15/03 Copyright 2003 Jeff Long 29
The CoRE HypothesisThe CoRE HypothesisWe can create “Competency Rule Engines” or CoREsWe can create Competency Rule Engines , or CoREs,
consisting of <50 ruleforms, that are sufficient to represent all rules found among systems sharing broad family resemblances e g all corporationsbroad family resemblances, e.g. all corporations. Their definitive deep structure will be permanent, unchanging, and robust for all members of the family, whose differences in manifest structures andwhose differences in manifest structures and behaviors will be represented entirely as differences in competency rules. The animation procedures for each engine will be relatively simple compared toeach engine will be relatively simple compared to current applications, requiring less than 100,000 lines of code in a third generation language.
4/15/03 Copyright 2003 Jeff Long 30
ReferencesReferences Long, J., and Denning, D., “Ultra-Structure: A design theory for
complex systems and processes.” In Communications of the ACM (January 1995)
Long, J., “A new notation for representing business and other rules.” In Long, J. (guest editor), Semiotica Special Issue on Notational Engineering, Volume 125-1/3 (1999)
Long, J., “How could the notation be the limitation?” In Long, J. (guest editor), Semiotica Special Issue on Notational Engineering, Volume 125-1/3 (1999)
Long, J., "Automated Identification of Sensitive Information in Documents Using Ultra-Structure". In Proceedings of the 20th Annual ASEM Conference, American Society for Engineering Management (October 1999)
4/15/03 Copyright 2003 Jeff Long 31
Recommended