1 “Star Wars” Revisited A Case Study In Ethics and Safety-Critical Software Professor Kevin W. Bowyer University of Notre Dame Copyright, Kevin W. Bowyer,

1

“Star Wars” RevisitedA Case Study In Ethics

and Safety-Critical Software

Professor Kevin W. BowyerUniversity of Notre Dame

Copyright, Kevin W. Bowyer, 2000,2001,2006,2007. All Rights Reserved. (Rev. 2/6/07)

2

Ballistic Missile Defense

A defensive system – to counter enemy offensive weapons.

Ballistic missiles follow a predictable trajectory.

3


The U.S. has a long history of BMD research and systems.

For example, the “Safeguard” system (1960s-70s) was meant to protect U.S. launch sites from pre-emptive attack.

4


The most important software elements of a BMD system are command and control.

Software development begins with analysis leading to requirements and specifications.

5

Software Life CycleSystem

Engineering

Analysis

Design

Coding

Testing

Maintenance

6

Software Life Cycle

System engineering –

Analyze the overall system,Specify system requirements, Decide on elements of system to implement in software.

7

Software Life Cycle

Software requirements –

Specify required functionality, performance, and interfaces for the software elements.

8

President Reagan’s SDI

In March 1983, President Reagan called for a “Strategic Defense Initiative” (SDI).

The SDI program came to be popularly called “Star Wars.”

9


10

Pause for Analysis

What overall requirements can you derive for the SDI system from Reagan’s speech?

(summarize in one paragraph)

11


“I call upon the scientific community … to give us the means of rendering these nuclear weapons impotent and obsolete.”

12


“I am directing … to define a long-term R&D program to begin to … eliminate the threat posed by strategic nuclear missiles.”

13


Which quote from the speech drives your requirements for the SDI system?

14

SDI Computing Task

The software must make the sensors, weapons, control, and communications elements of the system work together to satisfy requirements.

15

SDI Computing Panel

A panel of experts studied the SDI computing problem(at $1,000 / day consulting!).

David Parnas resigned from the panel and declared the software to be impossible.

16

1985 CPSR-MIT DebateThe Computing Professionals for Social Responsibility (CPSR) sponsored a debate at MIT, featuring members of the SDI computing panel.

17

1985 CPSR-MIT Debate

Michael Dertouzos, moderatorDavid Parnas, against SDI(Joseph Weizenbaum, against)

Charles Seitz, for SDI(Danny Cohen, for)

18

Michael Dertouzos, moderator

19

Pause for AnalysisBased on Dertouzos’ overview, what more can be said about the computing requirements for SDI?

(summarize in one page)

20

Michael DertouzosStar Wars – Can the computing requirements be met?

Assume that the sensors and weapons can be developed and put in space, are affordable, …

21

Michael DertouzosParameters of the problem –• watch 50 million km2 area,• track up to 3,000 missiles, • up to 30,000 warheads• up to 300,000 decoys,• it all happens in minutes.

22

Michael DertouzosElements of the task –• Form target tracks.

– Sensors and geographic-scale pixels– Consistent, distributed database.

• Allocate weapons to tracks.• Communicate with humans.• Withstand attack.

23

QuestionWhat is your first answer to

Dertouzos’ question –

“Star wars: can the computing requirements be met?”

24

QuestionWhy did some of the audience

laugh after Dertouzos mentioned “… consistent distributed database …?”

25

Consistent Distributed Database

“A database that consists of two or more data files located at different sites on a computer network. Because the database is distributed, different users can access it without interfering with one another.

However, the DBMS must periodically synchronize the scattered databases to make sure that they all have consistent data.”

http://www.webopedia.com/TERM/D/distributed_database.html

26

David Parnas, arguing against

27

Pause for Analysis

Sketch Parnas’ argument in premise-conclusion style: Since Premise, and Premise, … Therefore Conclusion.

(Hint: identify conclusion first.)

28

Parnas’ ConclusionWhich is a proper conclusionof his technical argument?

• U.S. should not pursue SDI.• SDI will make U.S. weaker. • It is not possible to build

trustworthy SDI software.

29

Parnas’ ConclusionIt is not possible to build trustworthy SDI software.

“trustworthy” = reliability similar to that of your car starting in the morning.

30

Parnas’ PremisesSince:• Specifications cannot be known

in advance, (because enemy controls factors

such as target / decoy features, attack load and structure, …)

31

Parnas’ PremisesSince:• Specifications not known in advance, • Realistic testing is essentially

impossible, (because, for example, link /

node failures under attack are not known in advance)

32

Parnas’ PremisesSince:• Specifications not known in advance, • Realistic testing is not possible,• Hard real-time deadlines do

not allow repair during use (attack is over in minutes)

33

Parnas’ PremisesSince:• Specifications not known in advance, • Realistic testing is not possible,• No chance to fix software during use,• No foreseeable advance in

software tech changes this (not language, methodology, …)

34

Parnas’ ArgumentSince:• Specifications not known in advance, • Realistic testing is not possible,• No chance to fix software during use,• No foreseeable technology changes this,

Therefore – It is not possible to construct SDIsoftware that you could trust to work.

35

1985 CPSR-MIT Debate

Michael Dertouzos, moderatorDavid Parnas, against SDI(Joseph Weizenbaum, against)

Charles Seitz, for SDI(Danny Cohen, for)

36

Charles Seitz, arguing for

37

Pause for Analysis

Sketch Seitz’ argument in premise-conclusion style: Since Premise, and Premise, … Therefore Conclusion.

(Hint: identify conclusion first.)

38

Seitz’ Conclusion

It is possible to create reliable SDI software.

39

Seitz’ PremisesSince• A hierarchical architecture

seems best, (because more natural, used in

nature, understood by military, allows abstraction up levels …)

40

Seitz’ PremisesSince• A hierarchical architecture seems best,• Physical organization should

follow logical organization, (simplest choice, natural)

41

Seitz’ PremisesSince• A hierarchical architecture seems best,• Physical organization also hierarchical,• Tradeoffs to make software

problem tractable are in the choice of system architecture

(not in new / radical methods)

42

Seitz’ PremisesSince• A hierarchical architecture seems best,• Physical organization also hierarchical,• This makes software problem tractable, • Loose coordination allows us to

infer system performance (assume stat. independence, …)

43

Seitz’ ArgumentSince• A hierarchical architecture seems best,• Physical organization also hierarchical,• This makes software problem tractable,• And allows system reliability estimate,

Therefore – It is possible to create reliable SDI battle management software.

44

Pause for AnalysisWhose argument is better?Why?

Do they start with the same problem definition?

45

David Parnas, Rebuttal

46

Charles Seitz, Rebuttal

47

Pause for AnalysisRelevant analogies to SDI?Why / why not?

• Space shuttle software• Telephone system software• Nuclear plant software• others?

48

Pause for Analysis

Outline the most realistic SDI software testing that you can.

49

Pause for AnalysisHow did you account for …

• real-world sensor inputs• variable weather conditions• target / decoy appearance• variable attack structure• attacked components failing

50

Fault Tolerant Software?

James Ionson, in “Reliability and Risk,” a CPSR video.

51

Fault Tolerant Software?“It is not error-free code, it is fault-tolerant code. And if another million lines has to be written to ensure fault-tolerance, so be it.”- James Ionson

52

Fault Tolerant Software?Diagram in premise-conclusion form the argument being made by James Ionson.

Does the argument make sense?Why / why not?

53

“Star Wars” TodayCurrent SDI-like programs are called “National Missile Defense.”

There are some potentially important differences.

54

“Star Wars” Today“One of the remarkable aspects of

the evolution of missile defenses is that few policy makers question the fundamental ability … to be effective. Instead they focus on timing, cost, ….”

(Mosher, page 39, IEEE Spectrum, 1997)

55

“Star Wars” Today“This is a sharp change from the Reagan years, perhaps because the technology used is closer at hand and the threats are smaller.”


56

Pause for AnalysisHow fundamentally does it change Parnas’ argument if the anticipated attack uses fewer and simpler missiles?

57

Parnas’ ArgumentHow are the premises changed?

• Specifications not known in advance.• Realistic testing is not possible.• No chance to fix software during use.• No foreseeable technology changes this.

None are changed “in principle” butit somehow more possible. Why?

58

“Star Wars” Testing“In the last 15 years, the U.S. has

conducted 20 hit-to-kill intercepts, …. Six intercepts were successful; 13 of those test were done in the last five years, and among them three succeeded.”


59

“Star Wars” Testing“No real attempts have been made

to intercept uncooperative targets – those that make use of clutter, decoys, maneuver, anti-simulation, and other countermeasures.”


60

“Star Wars” Testing“Test … of a powerful laser has been blocked by … bad weather and software problems.

… a software problem caused the laser to recycle, or unexpectedly lose power ….”

(R. Smith, Washington Post, Oct 8, 1997)

61

Schwartz versus TRWIn 1996, ex TRW engineer Nira Schwartz filed a “False Claims Act” suit, alleging that results of tests to distinguish warheads and decoys were falsified by TRW.

(featured on “60 Minutes II” in January 2001)

62

Schwartz versus TRWSchwartz claims – that TRW “knowingly made false test plans, test procedures, test reports and presentations to the government … to remain in the program.”

63

Schwartz versus TRWSchwartz claims – “I say to my boss, “It is wrong, what we are doing; it is wrong.” And the next day, I was fired.”

64

Schwartz versus TRWTRW says – “TRW scientists and engineers devoted years to this complex project, while Ms. Schwartz, in her six months with the company … Her understanding … is insufficient to lend any credibility to her allegations.”

65

Schwartz versus TRWDOD criminal investigator says – “absolute, irrefutable, scientific proof that TRW’s discrimination technology does not, cannot, and will not work” …TRW “knowingly covering up.”

66

Schwartz versus TRWDOD panel then said – “TRW’s software and sensors are “well designed and work properly” provided that the Pentagon does not have any wrong information about what kind of warheads and decoys an enemy is using.”

67

Schwartz versus TRWLt. General Kadish – “Right now, from what I see, there is no reason to believe that we can’t make this work. But there’s a lot more testing to be done.”

68

Schwartz versus TRWCongressman Curt Weldon, R-PA:“If we don’t build a new aircraft carrier, we have older ones. If we don’t build a new fighter plane, we have older ones. If we don’t build missile defense, we have nothing.”

What is the premise-conclusion summary of this argument?

69

Schwartz versus TRWCongressman Curt Weldon, R-PA:On 50 Nobelists’ anti-BMD letter - “I don’t know any of them that’s come to Congress or me. I mean … its easy to get anyone to sign a letter. I sign letters all the time.”


70

Schwartz versus TRWCongressman Curt Weldon, R-PA:“There were scientists that who made the case against Kennedy that it was crazy, we’d never land on the moon. And I characterize Postol now as one of those people.”


71

Ethical IssuesWhat are some of the important ethical questions?

And what guidance do the codes of ethics give on these questions?

72

Ethical Issues• How to interact with colleagues

with whom you disagree?

• When to blow the whistle?

• Should you accept work on an “impossible” but $$$ project?

73

Dealing with ColleaguesAITP Standards of Conduct:

“In recognition of my obligation to fellow members and the profession I shall cooperate with others in achieving understanding and in identifying problems.”

74

Dealing with ColleaguesItem 5.12 of ACM / IEEE-CSSoftware Engineering Code:

“Those managing or leading software engineers shall not punish anyone for expressing ethical concerns about a project.”

75

Accept Impossible Work?Item 3.2 of ACM / IEEE-CSSoftware Engineering Code:

“Software engineers shall ensure proper and achievable goals and objectives for any project on which they work or propose.”

76

Accept Impossible Work?Item 1.3 of the ACM / IEEE-CSSoftware Engineering Code:

“Software engineers shall accept software only if they have a well founded belief that it is safe, meets specifications, passes appropriate tests, …”

77

Blow the Whistle?AITP Standards of Conduct:

In recognition of my obligation to society, I shall never misrepresent or withhold information that is germane to a problem or situation of public concern nor allow any such known information to remain unchallenged.

78

Blow the Whistle?Item 1.4 of ACM / IEEE-CS Software Engineering Code:

“Software engineers shall disclose to appropriate persons or authorities any actual or potential danger to the user, the public … that they reasonably believe …”

79

Summary• Difficult ethical issues arise in

creation of safety-critical software.

• Trustworthy SDI software is more clearly impossible in retrospect.

• Modern, smaller SDI-like programs appear more tractable.

80

National Science Foundation grant DUE 97-52792

Thanks to

for partial support of this work.

81

Computing Professionals for Social Responsibility

(www.cpsr.org)

Thanks to the

for permission to distribute digitized video of the debate.

82

Thanks to

for commenting on a draft of the paper describing this module.

David Parnas

Chuck Seitz

83

Thanks to the

for help in obtaining the video of Reagan’s 3/23/83 speech.

The Ronald Reagan Presidential Library

(www.reagan.utexas.edu)

84

Thanks to

for technical assistance.

Christine KranenburgLaura Malave

Melissa ParsonsJoseph Wujek

85

The End.

Documents

1 “Star Wars” Revisited A Case Study In Ethics and Safety-Critical Software Professor Kevin W. Bowyer University of Notre Dame Copyright, Kevin W. Bowyer,