Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University

Online Behavioral Experiments

with NodeGame

Stefano BaliettiNetwork Science Institute, Northeastern University

Harvard Institute for Quantitative Social Science

D'Amore McKim School of Business, Northeastern University

@balietti | @nodegameorg | stefanobalietti.com | nodegame.org

1. Become confident about the principles and the methodology

behind social science scientific experiments

2. Get a quick-start with the software nodeGame to conduct

experimental research online

Goals of the Workshop

Workshop Instructors


Me

Social Scientist Open Source

Developer


Me

Social Scientist

Open

Source

Developer

What is an experiment?

An experiment is a methodological procedure carried out with the

goal of verifying, falsifying, or establishing the validity of a hypothesis.

A test under controlled conditions that is made to demonstrate a

known truth, examine the validity of a hypothesis, or determine the

efficacy of something previously untried.

An experiment is an empirical method that arbitrates between

competing hypotheses.


Internal validity:

The extent to which a causal inference based on a study is

warranted by an experiment.


Control

3 Pillars of Experimental Design

RandomizationIndependence

Control


RandomizationIndependence

Deliberate choice of the variables

that will be kept constants and those

that will be varied: treatment variables

Control


Randomization

Independence

Treatment variables must not be correlated

Control


Randomization

Independence

Participants are randomly assigned to conditions

Conditions are executed in random order

Stratification might be required

Internal validity:



External validity:

The extent to which the results of a study can be generalized

or extended to others.


Internal validity:



External validity:

The extent to which the results of a study can be generalized

or extended to others.


External Validity

Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610)

External Validity


External Validity

Telescope:

From Land to Sky


External Validity

Telescope:

From Land to Sky

Experiments:

From Lab to Real Life

Super-brief history of scientific experiments

• Novum Organum (1620)

• Scientific induction

• Torturing Nature to extract her secrets

Super-brief history of scientific experiments (1/3)

• Novum Organum (1620)

• Scientific induction

• Torturing Nature to extract her secrets


Ever heard of Phlogistons??

“There is a property common to almost all

the moral sciences, and by which they are

distinguished from many of the physical;

that is, that it is seldom in our power to

make experiments with them”

John Stuart Mill, 1836


“There is no laboratory in which economists can test

their hypothesis”

, 1993

Super-brief history of scientific experiments (2B/3)

Nobel prize for Vernon Smith

“for having established laboratory

experiments as a tool in empirical

economic analysis, especially in the

study of alternative market mechanisms”

2002


The Puzzle of Cooperation

The Ultimatum Game

The Ultimatum Game

Rules:

• 2 Players (A,B)

• 100 $

The Ultimatum Game

Rules:

• 2 Players (A,B)

• 100 $

• Player A makes a proposal

The Ultimatum Game

Rules:

• 2 Players (A,B)

• 100 $


• Player B can accept or refuse the split.

The Ultimatum Game

Rules:

• 2 Players (A,B)

• 100 $



• If B accepts each one gets his/her

share.

The Ultimatum Game

Rules:

• 2 Players (A,B)

• 100 $




share.

• If B refuses both players get nothing.

The Ultimatum Game: How to Split 100$

Rules:

• 2 Players (A,B)

• 100 $




share.

• If B refuses both players get nothing.


A B


Proposer RespondentRationalRational


Proposer Respondent

I say 99 for me

and 1 for you.

RationalRational


Proposer Respondent

I say 99 for me

and 1 for you.

RationalRational

I say OK.


Human Being Human Being

I say 99 for me

and 1 for you.


Human Being Human Being

I say…$!!@**I say 99 for me

and 1 for you.

Nowak (2006) 5 rules for Cooperation

Traditional Laboratory Research

• Decision making tools, heuristics, models of human behavior



• BUT



• BUT

• Small groups of individuals, generally 4-12 (bounded by uni. Lab)



• BUT


• Behavior of individuals in social networks under-researched



• BUT



• Limited in the number of hypotheses that can be tested



• BUT




• Observations are limited in time



• BUT





• Limited(?) external validity



• BUT





• Limited(?) external validity

• Generally high costs for an academic budget

The Present/Future of Behavioral Research

The Present/Future of Behavioral Research


Me

Social Scientist Open Source

Developer


Me

Social Scientist

Open Source Developer

nodeGame 3.x

Install nodeGame 3.x

1. Go to http://nodegame.org

2. Click on the “Get Started” button

3. Follow instructions to install Development version

4. Keep following my talk!

http://nodegame.org/

nodeGame 3.x Architecture Overview




~46.000 lines

(including comments)

Launch nodeGame 3.x

Open a terminal and browse to the nodegame folder

Start the server with the command: node launcher.js

Open a browser and access the default ultimatum game at the url:localhost:8080/ultimatum

Start an automated player (bot) at the url:localhost:8080/ultimatum?clientType=autoplay

Access the administrator interface at the url:localhost:8080/ultimatum/monitor/

Create a New Game

cd nodegame/bin/

node nodegame create-game gamename <author> <email>

Creates a new game in the games/ folder (usually nodegame/games/)

Important! Windows users must install the nodegame-generator package

manually from the git repository (or npm)

git clone https://github.com/nodeGame/nodegame-generator.git

cd nodegame-generator

npm install

Anatomy of a Game

Anatomy of a Game

Features:- Define game variables and treatments

- Define the game sequence with the Stager API

- Implement the game sequence into client types

The Game Sequence and Client Types

Game Sequence Client Typesstager.stage("instructions")

.step("instructions_1")


.step("instructions_3");

stager.stage("quiz");

stager.repeat("game", 3)

stager.stage("questionnaire");

stager.extendStage("game", {

steps: ["offer", "respond",

. "display_results"]

});

Code Snippet


Game Sequence Client Types

stager.stage("instructions")



.step("instructions_3");

stager.stage("quiz");

stager.repeat("game", 3)

stager.stage("questionnaire");


steps: [ "offer", "respond",

. "display_results" ]

});

Code Snippet



Client types implement the game sequence

by extending its steps and stages.






Game Sequence Client Types Code Snippet

stager.setDefaultProperty("coins",100);


coins: 200

});

stager.extendStep("offer", {

coins: 50

});


Game Sequence Client Types Code Snippet

stager.setDefaultProperty("coins",100);


coins: 200

});

stager.extendStep("offer", {

coins: 50

});

node.game.getProperty("coins") returns 50 in

step 'offer', 200 in steps 'respond' and 'display results'

inside the 'game' stage, and 100 in any other step.

Step/Stages Properties are hierarchical

Some Built-in Step-Properties

Property Description Type

init Executed upon entering the step/stage function

exit Executed upon exiting the step/stage function

timer Sets the maximum execution time for step (ms) number

timeup Executed if the timer expires function

done Executed when the step terminates function

cb The actual step execution function

Logic Client Type

Executed on the server, governs a game room

- Makes groups

- Keep track of game history

- Handles disconnections and reconnections

- Access to:- internal Database NDDB: node.game.memory

- player list: node.game.pl

- channel and registry object

Player Client Types

Executed on a client, e.g. a browser

- Display/manipulate information on screen

- Exchange messages with the server and other players

- Respond to user events

- Determine when the current step is finished (“done”)

W (Window): manipulates information on screen and the behavior of user interface,

e.g. disabling right click, locking screen, etc.

At the end of each step, the game developer must call the methodnode.done, which triggers a procedure that advances the client to the next step.

Inside a Step Callback Function

Instructions Step

- Player client type

- No interaction with other players

- No interaction with the page

- No time limit

- Simply click “Done” when

finished reading and advance to

next step


Instructions Step




- No time limit



next step


stager.extendStep('instructions_1', {

frame: "instr1.htm"

});

Instructions Step




- No time limit



next step



frame: "instr1.htm"

});

Widget Step


- No interaction with other players (at

the moment)

- Standardized interaction

- With or without time limit

- Register the mood of the participant

using the PANAS method.

Instructions Step




- No time limit



next step



frame: "instr1.htm"

});

Widget Step



the moment)





Instructions Step




- No time limit



next step



frame: "instr1.htm"

});

Widget Step



the moment)





Instructions Step




- No time limit



next step


widget: "MoodGauge"

});

Roles (version 3.5 dev)

stager.extendStep('bidder', {

matcher: {

// Available roles.

roles: [ 'BIDDER', 'RESPONDENT'],

// Each player is matched with a

// partner and is assigned a role.

match: 'roundrobin',

// How to continue matching after

// all combinations are exhausted

cycle: 'repeat_invert'

}

});

Logic Client Type

Roles (version 3.5 dev)


roles: {

BIDDER: {

frame: 'bidder.htm',

cb: function() {

// Bidder stuff.

}

},

RESPONDENT: {

frame: 'resp.htm',

cb: function() {

// Respondent stuff.

}}}

});

Player Client Type


matcher: {

// Available roles.

roles: [ 'BIDDER', 'RESPONDENT'],

// Each player is matched with a

// partner and is assigned a role.

match: 'roundrobin',

// How to continue matching after

// all combinations are exhausted

cycle: 'repeat_invert'

}

});

Logic Client Type

Inside the Step Callback Function

// The respondent receives the offer,

// displays it, and terminates the step.

// On incoming data labeled "offer",

// executes the function.

node.on.data("offer", function(msg) {

// Display offer using W.

W.setInnerHTML("offer", msg.data);

// End step.

node.done();

});

// Player made a choice

// and clicked the submit button.

submit.onclick = function() {

// Send offer to respondent.

node.say("offer", "respondent", 1);

// End step, store offer in server.

node.done({ offer: 1 });

};

I say 99 for me

and 1 for you.

Proposer Respondent

NDDB: Internal Database

// The respondent receives the offer,

// displays it, and terminates the step.

// On incoming data labeled "offer",

// executes the function.

node.on.data("offer", function(msg) {

// Display offer using W.

W.setInnerHTML("offer", msg.data);

// End step.

node.done();

});

// Player made a choice

// and clicked the submit button.

submit.onclick = function() {

// Send offer to respondent.

node.say("offer", "respondent", 1);

// End step, store offer in server.

node.done({ offer: 1});

};

I say 99 for me

and 1 for you.

Proposer RespondentLogic Client Type

NDDB database

node.game.memory



// Set an object in memory explicitly.

node.set({ foo: "bar" });


{

stage: {

stage: 2,

step: 1,

round: 1

},

player: "respondent",

timestamp: 1480336721939

offer: 1, // foo: "bar"

// node.done specific.

timeup: false,

time: 4321, // from step.

done: true

};





var m = node.game.memory; // NDDB instance.

// Lazy loading.

m.select("player", "=", "respondent");

m.and("stage.step", "=", 1)

.and("offer")

.fetch()[0];

// Use indexes.

m.player["respondent"].last();

m.stage["2.1.1"].select("offer").fetch()[0];




{

stage: {

stage: 2,

step: 1,

round: 1

},

player: "respondent",

timestamp: 1480336721939

offer: 1, // foo: "bar"

// node.done specific.

timeup: false,

time: 4321, // from step.

done: true

};


// Modify database items upon inserting them.

node.game.memory.on("insert", function(item) {

// Decorate item or save it somewhere else.

});

// Disable internal database completely.

// Remove default event listener.

node.off('in.set.DATA');

// Add new one.

node.on('in.set.DATA', function(msg) {

// Save msg somewhere else, e.g. Mongo.

});

Event Listener

Manager

node.game.events


// Modify database items upon inserting them.

node.game.memory.on("insert", function(item) {

// Decorate item or save it somewhere else.

});

// Disable internal database completely.

// Remove default event listener.

node.off('in.set.DATA');

// Add new one.

node.on('in.set.DATA', function(msg) {

// Save msg somewhere else, e.g. Mongo.

});

Event Listener

Manager

node.game.events

Events in nodeGame 3.x

Internal: generated by the game or by the nodeGame engine.

Incoming: triggered by incoming messages, 'in.[target].[action]', ‘in.set.DATA’




// Registering an internal event listener.

node.on('LOCAL_EVENT', function(a, b) {

// This is local event.

});

// Triggering a local event.

node.emit('LOCAL_EVENT', 1, 2);




// Registering an internal event listener.

node.on('LOCAL_EVENT', function(a, b) {

// This is local event.

});

// Listening for incoming DATA msgs.

node.on('in.say.DATA', function(msg) {

// I got a data msg!

});

// Listening for incoming "foo" DATA msgs.

node.on.data('foo', function(msg) {

// I got a data msg labeled "foo"!

});


node.emit('LOCAL_EVENT', 1, 2);


node.say('foo', "bar");


Events are hierarchically organized, where you register your event matters!






});






});

Valid only in Quiz stage






});

Registered in init or cb function of Step 3.3

Valid only in step 3.3 (any round).






});

Registered in the init function of “Game”

Valid throughout all steps of “Game” stage



How to register a listener valid throughout all

the game?



stager.setInitCb(function() {




});

});

How to register a listener valid throughout all

the game?

In the init function of the game!

Anatomy of a Game

Topics Covered and To Cover

Topics Covered and To Cover

Recruiting Participants

Free Platforms

Psychological Research on the Net http://psych.hanover.edu/research/exponnet.html

The Web Experiment List http://www.wexlist.net

Online Social Psychology Studieshttp://www.socialpsychology.org/expts.htm

Volunteer Sciencehttp://www.volunteerscience.com

Recruiting Participants

Commercial Platforms

ODesk, Freelancer, Elance, Guru, CrowdFlower, Innocentive, SurveySampling,

UpWork …

Most used in research:

How To Run a Game on AMT:

The Big Picture

1. Upload nodeGame server + game to a servere.g. Heroku cloud: https://github.com/nodeGame/nodegame-heroku

2. Create a new HIT on Amazon Mechanical Turk (AMT), e.g. using the requester

interface or other service API.

3. Use the nodegame-mturk package https://github.com/nodeGame/nodegame-mturk/

to handle:

- Connection and authentication from AMT HIT page to nodeGame server.

- Opening/expiring/extending HIT while running it.

- Payments, bonuses and qualifications after HIT is done.

https://github.com/nodeGame/nodegame-heroku

https://github.com/nodeGame/nodegame-mturk/

Known Issues Conducting

Synchronous Experiments on AMT

1. Not a survey or classification task, need to have DIFFERENT

people at the same time on your platform





2. Btw, not too few, not too many






3. Waiting time in the waiting room should be limited and

homogenous across participants








4. Demographics should be stratified as much as possible









5. Participants should not re-take the same experiment again










6. Dropouts rate (non-completing the HIT) signigicantly affects the

whole experiment, increasing costs $$ and time to complete it





Pay minimum wage (at least). Make bonus dependent on

performance






Rule of thumb: 2.8 times desidered number of participants

Expire HIT as soon as desired number of participant entered

the experimental pipeline (+small buffer)

3x








Rule of thumb: 2.8 times desidered number of participants

Expire HIT as soon as desired number of participant entered

the experimental pipeline (+small buffer)

Dispose participants who waited too long (compensated)




Waiting room can group players based on features

Collect some demographics features in a pre-experimental part

Start 2 (or more) groups at the same time




Create one HIT (no group) with 3x rule

Keep track of turker ids (nodegame-mturk)

Assign qualification via AMT API (nodegame-mturk)



6. Rate of dropouts (non-completing the HIT) significantly affects the


• Make sure waiting time is not too long (previous advices)






• Inform about required time BEFORE and DURING experiment







• Design experiments that are robust to dropout (as much as

possible). For example, use shares instead of fixed quantities.









• Use requirements checkings.









• Use requirements checkings.

• Use “seriousness checks” and “warm-up” phases.

How To Run a Game on AMT

AMT

How To Run a Game on AMT

AMT HIT

Not Just Experiments,

also Gamification

• Industry: lots of A/B testing and customer surveys

• Gamification: applies elements of game playing to solve

complex and simple problems

• Increasing trend

• Improve engagement and understanding

• Fun

• Generate More and Better Ideas

• Train Machine Learning Algorithms

(http://quickdraw.withgoogle.com)

http://quickdraw.withgoogle.com/

Contributions Welcome!

• Using it, request features, report bugs

• Help developing

• Extremely well documented source code

Contributions Welcome!

• Using it, request features, report bugs

• Help developing

• Extremely well documented source code

Thank You!

@balietti | @nodegameorg | stefanobalietti.com | nodegame.org

Technology

Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University