29
Cover Page Uploaded June 27, 2011 Towards a New Paradigm to Resolve the Software Crisis Author: Jeffrey G. Long ([email protected] Date: February 5, 20033 Forum: Talk presented at the University of North Carolina, Chapel Hill. Contents Page 1: Proposal Pages 228: Slides (but no text) for presentation License This work is licensed under the Creative Commons AttributionNonCommercial 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/bync/3.0/ or send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.

Towards a new paradigm to resolve the software crisis

  • View
    603

  • Download
    0

Embed Size (px)

DESCRIPTION

February 5, 2003: “Towards a New Paradigm to Resolve the Software Crisis”. Talk given at University of North Carolina, Chapel Hill.

Citation preview

Page 1: Towards a new paradigm to resolve the software crisis

Cover Page 

Uploaded June 27, 2011 

 

Towards a New 

Paradigm to Resolve 

the Software Crisis Author: Jeffrey G. Long ([email protected] 

Date: February 5, 20033 

Forum: Talk presented at the University of North Carolina, Chapel Hill.

 

Contents 

Page 1: Proposal 

Pages 2‐28: Slides (but no text) for presentation 

 

License 

This work is licensed under the Creative Commons Attribution‐NonCommercial 

3.0 Unported License. To view a copy of this license, visit 

http://creativecommons.org/licenses/by‐nc/3.0/ or send a letter to Creative 

Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA. 

Page 2: Towards a new paradigm to resolve the software crisis

Towards A New Paradigm to Resolve the Software Crisis Various kinds of metrics have been used to indicate the scope and magnitude of the software crisis. Some of these pertain to the high percentage of uninstalled systems; others to the increasing length of the applications backlog in most shops; others to the high lifetime cost of developing and maintaining software, whether custom-made or packaged; and others to the opportunity costs. All of these metrics indicate that the software business continues to be in a state of crisis which persists in spite of technical advances such as database management systems; structured code; improved and more formal development practices; the use of packaged software, code generators, computer-aided software engineering tools; component-based, object-oriented, multi-tier software architectures; and colossal budgets. The problem becomes worse as society expects more from computers and becomes more dependent upon them. User and organizational frustrations with this problem have caused CIOs to be replaced regularly, and, increasingly, have resulted in a strong trend to simply outsource all MIS computer and software operations -- and blame. This talk presents a new paradigm that has been developed and tested since 1985. It is based on an analysis of the nature of rules, their underlying common structure, and how rules are and ought to be represented. The approach is called “Ultra-Structure”. Rules are typically hard-coded into software applications, and the maintenance of these rules as they change is the primary cause of subsequent software changes. One unfortunate consequence of this standard approach is that subject experts must somehow communicate the rules they wish to see automated to programmers who typically are not experts in the subject matter of the application; much is lost in the translation. Another unfortunate consequence is that the software becomes large, often millions of lines of code, such that no one involved in the project can really comprehend or manage it. There have been of course numerous initiatives directed at solving these problems, but the solutions have been ineffectual and the problems they address are actually secondary and symptomatic rather than primary. The basic premise of this new approach is that we can resolve all of the major difficulties with software development, and generate significant new kinds of benefits for users, by removing most rules and all knowledge of the world from software and instead representing them the same way we represent data, i.e. as tables in a relational database. This approach combines key features of the normally disparate areas of management information systems, expert systems, and simulations, borrowing the strengths of each and largely eliminating the known problems of each. The approach can be applied to any kind of rule-based system, whether the application is intended to support business, scientific, or other application areas. It has been applied thus far to business and natural language applications, and an early prototype of its applications to biology has been developed by Dr. Gidding’s laboratory.

Page 3: Towards a new paradigm to resolve the software crisis

Towards A NewTowards A New Paradigm to Resolve the

S f C i iSoftware Crisis

Jeff Long

Feb 5, 2003

[email protected]

Page 4: Towards a new paradigm to resolve the software crisis

P d A dProposed Agenda

C t di dCurrent paradigm and consequences thereof (15 minutes)

Proposed alternative paradigm and consequences thereof (15 minutes)

Technical analysis of rules and their implementation (15 minutes)p ( )

General Q&A (15 minutes)

2/5/03 Copyright 2003 Jeff Long 2

Page 5: Towards a new paradigm to resolve the software crisis

P diParadigm

A f l ki t bj tA way of looking at a subject

An example, pattern, archetype, or model

A set of unconscious assumptions we have about a subjecthave about a subject

2/5/03 Copyright 2003 Jeff Long 3

Page 6: Towards a new paradigm to resolve the software crisis

C t S ft P diCurrent Software Paradigm

All li ti d fi d i t fAll applications are defined in terms of algorithms and data

Algorithms are the rules which are used to manipulate datato manipulate data

The model for this is the abacus

2/5/03 Copyright 2003 Jeff Long 4

Page 7: Towards a new paradigm to resolve the software crisis

Algorithms are implemented as softwareAlgorithms are implemented as software Everything else, such as inventory quantities or employee names isquantities or employee names, is implemented as data

Thus most rules (e.g. work processes, decision trees) are implemented asdecision trees) are implemented as software

2/5/03 Copyright 2003 Jeff Long 5

Page 8: Towards a new paradigm to resolve the software crisis

I di t CImmediate Consequences

E li ti i l t fEvery application requires a lot of software, often >1 million lines

No one can really understand or manage this level of complexity

Th b i ftThere are numerous bugs in any software system

Th ft h t b h d bThe software has to be changed as bugs are found and eliminated

2/5/03 Copyright 2003 Jeff Long 6

Page 9: Towards a new paradigm to resolve the software crisis

Th i l t f “ i t ” f thThere is a lot of “maintenance” of the software

The software has to be changed as the rules change

R l h bRules change as users become more sophisticated

R l h d t t lRules change due to external pressures as well

2/5/03 Copyright 2003 Jeff Long 7

Page 10: Towards a new paradigm to resolve the software crisis

Th h t ft th lThe changes to software themselves introduce new bugs

Extensive testing is required in order to have any comfort level

E t i d t ti i b tExtensive documentation is necessary, but often skipped; maintenance programmers don’t know why system was so designeddon t know why system was so designed

2/5/03 Copyright 2003 Jeff Long 8

Page 11: Towards a new paradigm to resolve the software crisis

Additi l A tiAdditional Assumptions

Users know what they want and canUsers know what they want and can communicate that to programmers

Programmers who have experience in anProgrammers who have experience in an industry can understand user requirementsUser requirements can be documented in aUser requirements can be documented in a form that is less complex than the actual working system

Software can be designed the same way as any other complex technology

2/5/03 Copyright 2003 Jeff Long 9

Page 12: Towards a new paradigm to resolve the software crisis

R lt t D tResults to Date

Increasing application backlog (maybeIncreasing application backlog (maybe use open source software)2/6 development projects cancelled;2/6 development projects cancelled; additional 3/6 considered failuresTrue value of very good designers andTrue value of very good designers and programmers increasingly recognizedBugs have greater and greaterBugs have greater and greater consequences; viruses don’t help

2/5/03 Copyright 2003 Jeff Long 10

Page 13: Towards a new paradigm to resolve the software crisis

Hi h t f Chi f I f tiHigh turnover of Chief Information Officers (average tenure: 18 months)

Increasing use of packagesThis typically forces changes in work processes, or else requires expensive customization (>2x package price)

Increasing use of outsourcing upon surrender (35-45% by 2005)

2/5/03 Copyright 2003 Jeff Long 11

Page 14: Towards a new paradigm to resolve the software crisis

L L d?Lessons Learned?Subject experts cannot communicate allSubject experts cannot communicate all requirements to programmers

their expertise took many years to acquirep y y qtheir own understanding will evolve

Subject experts must see working prototypes, t t tinot paper representations

Subject experts must be able to directly and continuously update rules as neededcontinuously update rules as needed“Corporate” knowledge must be externalized

2/5/03 Copyright 2003 Jeff Long 12

Page 15: Towards a new paradigm to resolve the software crisis

A Alt ti P diAn Alternative Paradigm

Remove most rules from softwareRemove most rules from software

Represent rules in canonical form, as fdata in a small set of tables

Externalize “corporate” knowledge in l i l k l d brelational knowledgebase

Let subject experts manage data

2/5/03 Copyright 2003 Jeff Long 13

Page 16: Towards a new paradigm to resolve the software crisis

I di t CImmediate ConsequencesSoftware size is reduced by 2+ orders ofSoftware size is reduced by 2+ orders of magnitude

simpler to create, manage, understand, test, p , g , , ,document, and teachremaining software has no knowledge of the world; it provides basic control logic that knowsworld; it provides basic control logic that knows what tables to check in what order, how to resolve conflicts, etc.

S ft d l t t i ll dSoftware development team is very small and manageable

2/5/03 Copyright 2003 Jeff Long 14

Page 17: Towards a new paradigm to resolve the software crisis

“Corporate” knowledge is externalized and isCorporate knowledge is externalized and is in a form anyone can see and understand

Knowledge is actionable by the computer: i h ki d i ireasoning, error checking, decision support

Subject experts can enter, change, and otherwise manage rules directly withoutotherwise manage rules directly, without programmer assistance

Data no longer merely defines certain details of the overall system; it defines the identity of the system

2/5/03 Copyright 2003 Jeff Long 15

Page 18: Towards a new paradigm to resolve the software crisis

Programmers do not need to know orProgrammers do not need to know or understand all rules, just enough to determine the classes of rules and thedetermine the classes of rules and the proper animation proceduresSerious prototyping becomes feasible;Serious prototyping becomes feasible; communications with users improvesTesting & QA can be far more rigorousTesting & QA can be far more rigorousDocumentation can be more complete

2/5/03 Copyright 2003 Jeff Long 16

Page 19: Towards a new paradigm to resolve the software crisis

R lt t D tResults to DateCommercial system: 7x growth withCommercial system: 7x growth with $1K/month software expense

Other prototypes in language, biology, legal, p yp g g , gy, g ,games, and artificial life seem to confirm basic claims and expectations

Cl SEI Vi i “Th i h fCloser to SEI Vision: “The right software, delivered defect free, on time and on cost, every time ”

2/5/03 Copyright 2003 Jeff Long 17

every time.

Page 20: Towards a new paradigm to resolve the software crisis

A l i f R lAnalysis of Rules

St t t f l d d i fStatement of rules and device for executing them can be different; need

t b ft f b thnot be software for both

Rules can be reformulated into a canonical form of “If a and b and c... then consider x and y and z”

2/5/03 Copyright 2003 Jeff Long 18

Page 21: Towards a new paradigm to resolve the software crisis

“If” l if diti d“If” values specify conditions under which each rule is examined; these are

ll d “f t ” i Ult St t Thcalled “factors” in Ultra-Structure Theory (UST)

“Then consider” values specify additional criteria that must be considered; these are called “considerations” in UST

2/5/03 Copyright 2003 Jeff Long 19

Page 22: Towards a new paradigm to resolve the software crisis

F t b i k i thi dFactors become primary keys in third-normal form RDBMS

Alternate keys can be specified if useful

Control logic (“animation procedures”) g ( p )reads relevant rules, including rules about selecting rules, and carries out g ,specified actions

2/5/03 Copyright 2003 Jeff Long 20

Page 23: Towards a new paradigm to resolve the software crisis

O i f l “ l l ” l ( thOne informal, “molecular” rule (e.g. three strikes and you’re out in baseball) may be translated as many atomic rulestranslated as many atomic rules

Basic process is todefine what exists (existential rules)define what exists (existential rules)

define relations between these (network & authorization rules)

define processes (protocol & meta-protocol rules)

2/5/03 Copyright 2003 Jeff Long 21

Page 24: Towards a new paradigm to resolve the software crisis

Tens of thousands of rules can beTens of thousands of rules can be grouped into a small number of classes based on their syntax and semanticsbased on their syntax and semanticsThese classes can be managed easily by the tools of a RDBMSby the tools of a RDBMSDesign can proceed by iterative prototype; small prototypes can easilyprototype; small prototypes can easily evolve to necessary level of complexity

2/5/03 Copyright 2003 Jeff Long 22

Page 25: Towards a new paradigm to resolve the software crisis

I l t tiImplementation

S ti f l i t l d fiSeparation of rules into classes defines tables in the RDBMS; there is no practical limit on number of rows in a tablelimit on number of rows in a table

Referential integrity and field edits ensure that rules maintain integritythat rules maintain integrity

Queries, report writers and forms make access to rules easy for authorized usersy

2/5/03 Copyright 2003 Jeff Long 23

Page 26: Towards a new paradigm to resolve the software crisis

Th R l f H th iThe Ruleform Hypothesis

CComplex system structures are created by not-necessarily complex processes; and these processes are created by the animation of p ycompetency rules. Competency rules can be grouped into a small number of classes whose form is prescribed by "ruleforms". While the p ycompetency rules of a system change over time, the ruleforms remain constant. A well-designed collection of ruleforms can anticipate all logically p g ypossible competency rules that might apply to the system, and constitutes the deep structure of the system.

2/5/03 Copyright 2003 Jeff Long 24

y

Page 27: Towards a new paradigm to resolve the software crisis

Th C RE H th iThe CoRE Hypothesis

WWe can create “Competency Rule Engines”, or CoREs, consisting of <50 ruleforms, that are sufficient to represent all rules found among systems sharing p g y gbroad family resemblances, e.g. all corporations. Their definitive deep structure will be permanent, unchanging, and robust for all members of the family, g g, y,whose differences in manifest structures and behaviors will be represented entirely as differences in competency rules. The animation procedures for p y peach engine will be relatively simple compared to current applications, requiring less than 100,000 lines of code in a third generation language.

2/5/03 Copyright 2003 Jeff Long 25

g g g

Page 28: Towards a new paradigm to resolve the software crisis

Ultra-Structure is an Example of Notational Engineering

M bl i i t thMany problems in science, government, the arts, and engineering are caused by the way we represent themwe represent them

These problems cannot be resolved with more computing power or more moneymore computing power or more money

They require a new abstraction which can be the basis of a notational revolution

2/5/03 Copyright 2003 Jeff Long 26

Page 29: Towards a new paradigm to resolve the software crisis

R fReferencesLong J and Denning D “Ultra-Structure: A design theory forLong, J., and Denning, D., Ultra-Structure: A design theory for complex systems and processes.” In Communications of the ACM (January 1995)Long, J., “A new notation for representing business and other g, , p grules.” In Long, J. (guest editor), Semiotica Special Issue on Notational Engineering, Volume 125-1/3 (1999)Long, J., “How could the notation be the limitation?” In Long, J. ( t dit ) S i ti S i l I N t ti l(guest editor), Semiotica Special Issue on Notational Engineering, Volume 125-1/3 (1999)Long, J., "Automated Identification of Sensitive Information in Documents Using Ultra-Structure" In Proceedings of the 20thDocuments Using Ultra-Structure . In Proceedings of the 20th Annual ASEM Conference, American Society for Engineering Management (October 1999)

2/5/03 Copyright 2003 Jeff Long 27