Toward A Reasonable Toward A Reasonable Programmer StandardProgrammer Standard

Responsibility and Negligence in Software Design

Overview of Presentation It’s an ethical issue – ask questions; make

comments

Why do we need a definition?

New technologies = new legal issues

Software, though not new, is applied in new areas

What to think when software causes harm?

Why a Definition?Objection: you gain nothing by stating a

definition

Objection: I know in advance that your definition is wrong

Objection: Standard of negligence doesn’t apply to software

The Law and Negligence

“Failure to be sufficiently careful in a matter in which one has a moral responsibility to exercise care...” Online Ethics Center: The Online Ethics Glossary Center. (2003, March 21).

Classic example: leaving a rake on your walkway – kills the mailman

Failing to shovel snow; doctors who leave surgery tools in a patient; etc.

When is a person negligent?

The reasonable person standard: “A phrase used to denote a hypothetical person who exercises qualities of attention, knowledge; intelligence, and judgment that society requires of its members for the protection of their own interest and the interests of others.”National Association for Court Management: Glossary of Terms.

Invokes different requirements in different societal roles – the reasonable doctor is very different from the reasonable homeowner

What is a reasonable programmer?

Test Case 1: Freeware Software is distributed freely and

with a carefully worded license

What’s the worst that can happen? Computer crashes -> data loss

Holding a programmer responsible for the data is too demanding – not everything is negligent

                                                                          

Test Case 2: Free Algorithm

Programmer posts implementation of a sorting algorithm

Algorithm doesn’t work on negative numbers

Company uses algorithm in air traffic control software

Moral intuition: company’s fault, not the algorithm programmer; notion of direct responsibility

Test Case 2: Ethical Analysis

Company had contractual and moral obligation to test their software sufficiently

Programmer had no intention of using algorithm in critical environment – no obligation

Conclusion: expectations of performance derive from moral and contractual obligation

Test Case 3a: Virus JunkieA programmer Q gets high off of

creating viruses

Q unwittingly unleashes virus, costing U.S. companies millions

Should Q be held responsible?

Test Case 3b: Router A properly functioning antivirus

program at company C would have stopped Q’s virus

Is C in any way responsible for the harm caused by the virus?

Test Case 4: Therac-25 Atomic Energy of Canada Limited: faulty software

leads to overdose of radiation -> death

AECL probably was ‘sufficiently careful’ – post-accident review showed many hours of testing

Contrasted with bug that fails to save internet books mark: serious consequences important to definition, as is amount of testing proportional to potential harms

Test Case 5: Patriot Missile Programming flaw -> system operates over 20 hours =

failure -> military issues patch, deployment is slow

Barracks destroyed, software patch arrives the next day

Raytheon’s fault? Testing revealed the flaw

Army’s fault? Slow deployment of patch, vague memo on proper operation, project extended beyond intended lifetime

New definition: direct causation of flaw, or indirect causation of conditions that lead to flaw

Final DefinitionUnreasonable, direct failure to be

sufficiently careful in software design and testing, or causation of such conditions leading to serious harm, wherein a programmer has a moral obligation to do so from a contractual or otherwise reasonable expectation.

Test Case 6: More VirusesLots of subtle issues, what can we

reasonably expect from each party?: Obviously, punish those who create virus OS designer let flaw propagate? Owners and designers of networks that

allow propagation? User who downloads attachment with

subject “Haven’t heard from you in a while!”?