A Popperian Platform for Programming and Teaching the Global Brain Karl Lieberherr Ahmed Abdelmeged...

A Popperian Platform for Programming and Teaching the

Global BrainKarl Lieberherr

Ahmed AbdelmegedNortheastern University, CCIS, PRL,

Boston

04/20/23 1

A claim is

• Meta information about one’s performance when interacting with another clever being.

• Meta information about the performance of one’s program.

4/24/2011 2Crowdsourcing

Outline

IntroductionIntroduction

Theory & hypothesesTheory & hypotheses

MethodsMethods

Results & analysisResults & analysis

ConclusionConclusion

Theory

Methods

Results

Conclusion

Introduction

04/20/23 3

Results Preview

• Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, tunedit.org, …

• SCG usage for teaching– Innovation Success with Undergraduates using SCG on

piazza.com: Qualitative Data Sources & Analysis– Avatar competitions are not for teaching (but for

competitive innovation)

• Theoretical Properties of SCG

04/20/23 4

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

SCG = Scientific Community Game = Specker Challenge Game

Specker

• Claims: – Specker(X,Y(X),f,c): ForAll x in X Exists y in Y:

f(x,y)≥c• Example:

– X = Conjunctive Normal Forms with restrictions– Y(X) = Assignments to CNFs– f(x,y) = fraction of satisfied clauses in x under y– c in [0,1]

04/20/23 5

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Popper

• That a claim is "falsifiable" does not mean it is false; rather, that if it is false, then some observation or experiment will produce a reproducible result that is in conflict with it.

04/20/23 6

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

What SCG helps with

• How to identify experts?

• How to decide if an answer is worthwhile?– Use scholars to choose the winners

• How to organize egoistic scholars to produce social welfare: knowledge base and know-how how to defend it.– The scholars try to reverse engineer the

solutions of winning scholars.

04/20/23 7

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Claims

• Protocol. Defines scientific discourse.• Scholars make a prediction about their

performance in protocol.• Predicate that decides whether refutation is

successful. Refutation protocol collects data for predicate.

• As a starter: Think of a claim as a mathematical statement: EA or AE.– all planar graphs have a 4 coloring.

04/20/23 8

Claim involving Experiment

Claim ExperimentalTechnique(X,Y,q,r)I claim, given raw materials x in X,I can produce product y in Y of quality qand using resources at most r.

9Crowdsourcing4/24/2011

Who are the scholars?

• Students in a class room– High school– University

• Members of the Gig Economy– Between 1995 and 2005, the number of self-

employed independent workers grew by 27 percent.

• Potential employees (Facebook on kaggle.com)• Anyone with web access; Intelligent crowd.

04/20/23 10

Kaggle.com Competitions2012

• Facebook recruiting competitions– Task: Data scientist– Reward: Job– Teams: 197

• Heritage Health Prize– Task: Hospital admissions– Reward: $ 3 million– Teams: 1118

• Chess ratings – Elo versus the Rest of the World– Task: Predict outcome of chess games– Reward: $ 617 – Teams: 257

04/20/23 11

Kaggle.com Competitions2012

• Eye Movements Verification and Identification– Task: Identify people– Reward: Kudos– Teams: 51

• EMC Data Science Global Hackathon– Task: Air Quality Prediction– Reward $ 7030– Teams: 114

04/20/23 12

What Scholars think about!

• If I propose claim C, what is the probability that– C is successfully refuted– C is successfully strengthened

• If I try to refute claim C, what is the probability that I will fail.

• If I try to strengthen claim C, what is the probability that I will fail?

1304/20/23

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Degree of automation with SCG(X)

14

no automationhuman plays

full automationavatar plays

degree of automation used by scholar

some automationhuman plays

0 1

more applications:test constructive knowledge

transfer to reliable, efficient software

avatar Bobscholar Alice

04/20/23

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Organizational Problem Solved

• How to design a happy scientific community that encourages its members to really contribute.

• Control of scientific community– tunable SCG rules– Specific domain, claim definition to narrow scope.

04/20/23 15

happy = no scholar is ignored.

What is a loose collaboration?

• Scholars can work independently on an aspect of the same problem.

• Problem = decide which claims in playground to oppose or agree with.

• How is know-how combined? Using a protocol.– Alice claimed that for the input that Alice provides,

Bob cannot find an output of quality q. But Bob finds such an output. Alice corrects.

– Bug reports that need to be addressed and corrections.

04/20/23 16Playground = Instantiation of Platform

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Theory

• Extensive Form Representation of Game• Community Property: All faulty actions can be

exposed.

04/20/23 17

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Extensive-form representation

1. the players of a game: 1 and 22. for every player every opportunity they have

to move3. what each player can do at each of their

moves4. what each player knows for every move5. the payoffs received by every player for

every possible combination of moves

04/20/23 18

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

1

propose claim C from Claims

2

refute(C,1,2)

p(C, …)?(1,-1):(-1,1)

1 scholar2 scholar

strengthen attempt C’ => C

refute(C’,2,1)

agree attempt C

refute(C,2,1)

p(C’, …)?(1,-1):(-1,1) p(C, …)?(1,-1):(-1,1)

04/20/23

refute attempt C

refute(C, proposer,other)p(…)?(proposer,other): (proposer,other)s: successfulu: unsuccessful

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

p(C’, …)?(-1,1):(1,-1)

u:1 2 s:1 2

s:1 2 u:1 2

p(C, …)?(0,0):(1,-1)s:1 2 u:1 2

19

Refutation Protocol

• Collects data given to predicate p. Alternates.

refute(C,proposer,other)

p(C, …)?(1,-1):(-1,1)

claim payoff for proposerif p true (defense)

payoff for otherif p true (defense)

payoff for otherif p false (refutation)

payoff for proposerif p false (refutation)

other tries to makep false while proposertries to make p true.

p false means successfulrefutation.

p true means successfuldefense.

04/20/23 20

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Reinterpret Refutation

• Refutation leads to successful strengthening or successful agreement.

04/20/23 21

Essence of Game Ruleswithout Payoff

• scholars: 1, 2• LifeOfClaim(C) = propose(1,C) followed by

(oppose(1,2,C)|agree(1,2,C)).• oppose(1,2,C) = (refute(1,2,C)|

strengthen(1,2,C,C’)), where stronger(C,C’).• strengthen(1,2,C,C’) = !refute(2,1,C’).• agree(1,2,C) = !refute(2,1,C)

04/20/23 22

blamed decisions:propose(1,C)refute(1,2,C)strengthen(1,2,C,C’)agree(1,2,c)

Winning/Losing

• Scholar who first violates a game rule, loses.• If none violate a game rule: the claim

predicate c.p(1,2, …) decides.

04/20/23 23

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Game Rules for Playground

• All objects exchanged during protocol must be legal and valid.

• Each move must be within time-limit.

04/20/23 24

Example: Independent Set

• Alice = proposer, Bob = other.• Protocol / claim: AtLeastAsGood. Alice claims

to be at least as good as Bob at IS.– Bob provides undirected graph G.– Bob computes independent set sB for G (secret).– Alice computes independent set sA for G.– Alice wins, if size(sA) >= size(sB) (= p(sA,sB)).

04/20/23 25

More examples of Protocols

• Let f(x,y)=x*y+(1-x)(1-y^2)). Alice claims Math(0.61): Bob constructs an x in [0,1] and Alice construct a y in [0,1], and Alice guarantees that f(x,y)> 0.61. True claim but can be strengthened to 0.618.

• Alice claims Solar(RawMaterials,m,0.61). Bob constructs raw materials r in RawMaterials and Alice constructs a solar cell s in Solution from r using money m and so that efficiency(s)> 0.61.

04/20/23 26

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Community Property

• For every faulty decision action there exists an exposing reaction that blames the bad decision.– Reasons:

• We want the system to be egalitarian. – It is important that clever crowd members can shine and

expose others who don’t promote the social welfare of the community.

• Faulty decisions must be exposable. It may take effort.

04/20/23 27

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Community PropertyAlternative formulation

• If all decisions by Alice are not faulty, there is no chance of Alice losing against Bob.– if Alice is perfect, there is no chance of losing.

• If there exists a faulty decision by Alice, there is a chance of Alice losing against Bob.– egalitarian game

04/20/23 28

Summary: faulty decisions

1. propose(Alice,C),C=false2. propose(Alice,C),C=not optimum, C=true3. refute(Alice,Bob,C),C=true4. strengthen(Alice,Bob,c,cs),c=optimum5. strengthen(Alice,Bob,c,cs),c=false6. agree(Alice,Bob,c),c=false7. agree(Alice,Bob,c),c=not optimum, c=true

04/20/23 29

SCG Equilibrium

• Reputations of scholars are stable.• The science does not progress; bugs are not

fixed, no new ideas are introduced.• Extreme, desirable situation: All scholars are

perfect: they propose optimal claims that can neither be strengthened nor refuted.

3004/20/23

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Claims: convergence to optimum

3131

0

1

qualitystrengthening

correct valuation

over strengthening

true claims(defendable)

false claims(refutable)

04/20/23

Convergence

• if every faulty action is exposed, convergence is guaranteed.

04/20/23 32

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Methods

• Developed Platform SCG Court = Generator of teaching/innovation playgrounds– http://sourceforge.net/p/generic-scg/code-0/110/tree/GenericSCG/– Developed numerous playgrounds for avatars.

• Developed Algorithms Course using Piazza based on SCG Court experience– role of scholar played by humans – piazza.com: encourages students to answer each

other’s questions.

04/20/23 33

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Avatar Interface

• AvatarI – public List<Claim> propose(List<Claim> forbiddenClaims); – public List<OpposeAction> oppose(List<Claim>

claimsToBeOpposed); – public InstanceI provide(Claim claimToBeProvided); – public SolutionI solve(SolveRequest solveRequest);

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

04/20/23 34

Instance Interface

• InstanceI – boolean valid(SolutionI solution, Config config); – double quality(SolutionI solution);

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

04/20/23 35

InstanceSet Interface

• InstanceSetI – Option<String> belongsTo(InstanceI instance);– Option<String> valid(Config config); }}

04/20/23 36

Protocol Interface

• ProtocolI – double getResult(Claim claim, SolutionI[]

solutions, InstanceI[] instances); – ProtocolSpec getProtocolSpec(); – boolean strengthenP(Claim oldClaim, Claim

strengthenedClaim);

04/20/23 37

Claim Class, for all playgrounds

• Claim – public Claim(InstanceSetI instanceSet, ProtocolI

protocol, double quality, double confidence)

04/20/23 38

Protocol Library

• AsGoodAsYou.java• ExistsForAll.java• ForAllExists.java• Renaissance.java• Survivor.java

04/20/23 39

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Claim Kinds in SCG Court

• Claim C(instance, q)• Claim C(InstanceSet, q)• Claim

MaxResource(Algorithm,InstanceSet,n,ResExp)• Claim

MinResource(Algorithm,InstanceSet,n,ResExp)• Claim IAmTheBest(), AtLeastAsGoodAsYou()• Claims from predicate logic and some of its

generalizations (IF Logic)

04/20/23 40

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Results

• Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, Operations Research Competitions, tunedit.org, http://eterna.cmu.edu/ …

• SCG usage for teaching using forum– Innovation Success with Undergraduates using SCG on

piazza.com: Qualitative Data Sources & Analysis• Avatar competitions are not for teaching (but

good for competitive innovation)• Theoretical Properties of SCG

04/20/23 41

SCG = Scientific Community Game = Specker Challenge Game

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

1

propose claim C from Claims

2

refute(C,1,2)

p(C, …)?(1,-1):(-1,1)

1 scholar2 scholar

strengthen attempt C’ => C

refute(C’,2,1)

agree attempt C

refute(C,2,1)

04/20/23

refute attempt C

refute(C, proposer,other)p(…)?(proposer,other): (proposer,other)s: successfulu: unsuccessful

p(C’, …)?(-1,1):(1,-1)

u:1 2 s:1 2

s:1 2 u:1 2

p(C, …)?(0,0):(1,-1)s:1 2 u:1 2

High competition

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

42

1

propose claim C from Claims

2

refute(C,1,2)

p(C, …)? (0,0) :(0,1)

1 scholar2 scholar

strengthen attempt C’ => C

refute(C’,2,1)

agree attempt C

refute(C,2,1)

04/20/23

refute attempt C

refute(C, proposer,other)p(…)?(proposer,other): (proposer,other)s: successfulu: unsuccessful

p(C’, …)?(0,1): (0,0)

u:1 2 s:1 2

s:1 2 u:1 2

p(C, …)?(0,0): (1,0)s:1 2 u:1 2

Low competition

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

43

Competition Knob: minimum

• For each scholar– count claims that were successfully opposed

(refuted or strengthened)• encourages strong claims• gather information from competitors for free

– count claims that were not successfully agreed• Good for teaching

– students want minimum competition– good students want to build social capital and

help weaker students

04/20/23 44

Piazza Results

• Lower competition knob for teaching.• For optimization claims got significant

scientific discourse.• Playgrounds cannot have too many scholars,

otherwise they are overwhelmed.– about 5 is a good size– use hierarchical playgrounds: winning teams

compete again

04/20/23 45

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Piazza Results

• Do not give hints at solutions. This significantly decreased the amount of discourse taking place.

04/20/23 46

Gamification of Software Development and Teaching

• Want reliable software to solve a computational problem? Design a game where the winning team will create the software you want.

• Want to teach a STEM domain? Design a game where the winning students demonstrate superior domain knowledge.

Crowdsourcing

Doesn’t TopCoder already do this?

STEM = Science, Technology, Engineering, and Mathematics

474/24/2011

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Conclusions for Teaching

• Transition– refute: (1,-1):(-1,1) -> (0,0) :(0,1)– strengthen: (-1,1):(1,-1) -> (0,1): (0,0)– agree: (0,0):(1,-1) -> (0,0): (1,0)

• creates better playgrounds for learning by lowering competition and increasing teaching between scholars.

04/20/23 48

IntroductionIntroduction TheoryTheory MethodsMethods ResultsResults ConclusionConclusion

Conclusions

• Flexible use of SCG using a forum environment with threads and replies using optimization playgrounds is productive:– teams took turns leapfrogging each other

04/20/23 49