eric c. larson | eclarson.com

mobile health for the masses

Assistant Professor Computer Science and Engineering

health monitoring using mobile phones

> slide about me

Shwetak Patel James Fogarty Jon Froehlich

the health landscape

75% of all US healthcare spending is on chronic disease

source: center for disease control, 2014

state of health in the US

US infant mortality rate ranks ~23rd in the world, but 1st in delivery cost

source: national vital statistics reports, 2014

9 out of 10 of US doctors feel medications are overprescribed

source: american medical association, 2012

solution: more actionable data, better managed care,

preventative care, but with lower cost

one strategy: mhealth

what is mhealth?

mHealth (em-‘helth )an abbreviation for mobile health, a term used for the practice of medicine and public health supported by mobile devices

the promise of mHealth:eliminate doctor visits remote / automatic diagnosis equalize developing countries

stress check

fitness trainer

heart rate

current mhealth

telemedicine

remote training

stress check

telemedicinefitness trainer

heart rate

current mhealthremote training

~30,000 apps for health ~95% are for calorie counting & exercise ~5% are remote monitoring, wellness,

smoking cessation, references, etc.

yet to be a disruptive mHealth technology

mHealth sensing outside the clinic

compliance?cost?

privacy?data reliability?

phone as a sensorbaseline process

clinical quantity

sensor or

human

embedded sensors processing

estimated

accelerometer gyroscope barometric pressure temperature magnetometer /compass dual camera / flash 1+ microphones proximity sensor capacitive sensor gps motorized actuator wireless antenna (s)

compliance++; cost--;

data reliability?

what can the mobile phone sense with clinical accuracy?

pupillary response

lung functionjaundice

pupillary response

lung functionjaundice

Total Serum Bilirubin

Medical GoldStandard

Transcutaneous

TcB

Bilirubinometer

TcBNon-invasive

Correlates Well

$7000

Quick results

Screening tool for TSB

20

15

10

5

0

0.5 1 2 3 4 5 6 Age (days)In Hospital

Biliru

bin

(mg/

dL)

Newborn Bilirubin Levels

75th percentile

25th percentile

Visual Assessment • Parents • Many physicians • Traveling practitioners

In Hospital At Home

Screening Challenges

Tend to underestimate

bilirubin level in blood

jaundice levelblood draw

yellowness camera processing

estimated

bilicam

Shwetak PatelJim Taylor Lilian DeGreef Mayank Goel Jim Stout

Study Evaluation

100 newborn participants • <1 day old when

enrolled

Data collected by medical professionals using iPhone 4S

BiliCam

TSB (ground truth)

TcB (control)

3 - 5 days old

Noisy Data

Automatic Quality Control

✔ ✖ ✖

✖ ✖ ✖

Ideal Glare Overexposed

Occlusion Shadow Underexposed

Algorithm Overview

Bilirubin Estimate

Color Balance

Extract Features

Apply Regressions

400 500 600

Wavelength (nm)

Rela

tive

Bili

rubi

n

Abs

orpt

ion

Prob

abili

ty

RGB

CrCbY

YCbCr

a*b*

L*

L*ab

with & without flash

skin Gradient (of RGB channels)

Extract Features

Color Balance

Regression Ensemble

No

90th percentileYes

regressions agree

mean

Sigmoidal

LARS-Lasso Elastic Net

Support Vector Regressions

Encapsulated Neighbor

Random Forest Regression

Bilirubin Estimate

0

5

10

15

20

25

0 5 10 15 20 25

Results

TSB Ground Truth (mg/dl)

Estim

ated

Bilir

ubin

(mg/

dl)

0

5

10

15

20

25

0 5 10 15 20 25

BiliCamrank order 0.85 correlation

Results

TSB Ground Truth (mg/dl)

Estim

ated

Bilir

ubin

(mg/

dl)

0

5

10

15

20

25

0 5 10 15 20 25

TcBs correlate 0.75 - 0.93

BiliCamrank order 0.85 correlation

TcBrank order 0.92 correlation

Results

TSB Ground Truth (mg/dl)

Estim

ated

Bilir

ubin

(mg/

dl)

Interpretation20

15

10

5

0

high risk

high intermediate risk

low intermediate risk

low risk

Biliru

bin

(mg/

dL)

0.5 1 2 3 4 5 6 Age (days)

20

15

10

5

0

high risk

high intermediate risk

low intermediate risk

low risk

Biliru

bin

(mg/

dL)

0.5 1 2 3 4 5 6 Age (days)

Bhutani Nomogram

20

15

10

5

0

high riskhigh intermediate risk

low intermediate risk

low risk

Biliru

bin

(mg/

dL)

0.5 1 2 3 4 5 6

Age (days)

9 high risk cases based on TSB

Interpretation

20

15

10

5

0

high riskhigh intermediate risk

low intermediate risk

low risk

Biliru

bin

(mg/

dL)

0.5 1 2 3 4 5 6

Age (days)

BiliCam 2/9 missed high risk (22%)85% blood draws avoided

Interpretation

20

15

10

5

0

high riskhigh intermediate risk

low intermediate risk

low risk

Biliru

bin

(mg/

dL)

0.5 1 2 3 4 5 6

Age (days)

BiliCam 2/9 missed high risk (22%) 85% blood draws avoided

TcB 2/9 missed high risk (22%) 88% blood draws avoided

BiliCam is sufficient for newborn Jaundice screening, but it is unknown how user error affects reliability

Interpretation

Next steps: developing world

Kernicterus:+21+($8+mill)+

Hazardous+jaundice:++1158+

($50,000)+

Extreme+jaundice:++2,317+($20,000)+

Severe+jaundice:+35,000+($8,500)+

Phototherapy:+290,000++($1,000)+

Visible+jaundice:+3.5+million+

Births/year:+4.1+million+

In the US Middle- & low-income countries:

• 75,000 cases kernicterus/year

• 114,000 newborn deaths/year

• 65% newborn deaths from kernicterus

kernicterus 21

($8 million)

Bhutani et al. 2013, Pediatric Research 2010

pupillary response

lung functionjaundice

pupillary response drug impairment

concussion stroke

pain fatigue

cognitive disabilities arousal

cognitive load

cognitive load refers to the total amount of mental effort being used in the working memory

multiply these two numbers

3 6

measurement pupillometer

IR gaze trackers

$4000$1000

pupillary change

cognitive loadcontact

IR camera

visible iris camera regression

estimated

PupilWare

Suku Nair Sohail Rafiqi Mark Chatchai Ephrem Fernandez

pupilware cognitive load study

uniquely brighter than the iris [43]. This contrast makes it rela-tively easy to measure the pupil’s center. It is important to note that eye tracking in general does not require pupil size measure-ment; it only requires measuring the center of the pupil or center of the iris. Therefore much of the techniques used in eye tracking are not directly applicable to pupillary response measurement.

Starburst [42] is a pupil segmentation algorithm that uses a hybrid of feature based and model-based approaches. Starburst estimates eye center and iteratively grows the pupil region to find the pupil edge using ellipse fitting techniques and RANSAC to eliminate outliers [44]. Starburst then finds the inliers from the ellipse fit-ting iterations. These inliers are sent to a final ellipse-fitting algo-rithm and, finally, pupil diameter is found by averaging the el-lipse width and height. We use a modified version of the Starburst algorithm in PupilWare, modified to more appropriately find the pupil edge without the high contrast provided by infrared light.

In recent work, Wood and Bulling were able to track eye gaze from tablet cameras [45]. Their prototype, EyeTab, uses a model-based approach to estimate the gaze without an infrared light source. EyeTab uses means of gradient algorithm (where most of the image gradients meet) [46] to determine the center of the eye and then identify points on the limbus edge (i.e., the edge of the iris) to fit an ellipse model. While EyeTab is closely related to PupilWare, the iris center is used to infer a participant’s gaze and therefore EyeTab does not measure pupillary dilation, which clearly separates the contributions of our analyses. However, this work showed that head pose and lighting could be controlled and compensated using a mobile device’s camera, clearly influencing the design of PupilWare.

3 DATA COLLECTION To validate the design of PupilWare algorithm we conducted an IRB approved human subject study at Southern Methodist Uni-versity as shown in Figure 1. As part of the study we replicated one of the classics Cognitive psychology experiments namely Digit Span task. The test is used to artificially induce the cogni-tive load on the participant. This is one of the classic cognitive pupillometry tests [19][18][32] that was later repeated by Klingner [30] using eye-trackers. A user is presented with a spe-cialized, short duration task meant to artificially induce mental workload. At the beginning of the task, the baseline pupil size is captured. At the end of the task subjective rating and task comple-tion time (TCT) are used to validate the workload.

Each trial is started with stabilization of pupil for 5 seconds. Se-quences of digits are spoken aloud to the user at the rate of 1 number per second. After a short period participants orally report back the sequence. This test assesses how much the pupil diame-ter increases as participants memorize the digit and decreases as these digits are reported back. We also capture any errors that the users make. We control the difficulty level by the number of dig-its in the sequence, i.e., 5, 7, or 9 (four iterations of each diffi-cult). In addition to collecting the pupil size we also keep track of number of times participants make a mistake.

For ground truth, we use two accepted devices from the psychol-ogy community—the Neuroptics VIP-200 Pupillometer [27] and Gazept Remote Eye Tracker [47]. The Neuroptics VIP-200 is a portable, battery operated, hand-held device that accurately measures pupil size with a resolution of 0.1 mm. This is same as used by ophthalmologist during an eye exam. The device can measure pupil diameter either with no background illumination for 2 seconds or variable light levels in one sequence for a total of

10 seconds. It reports average pupil size and standard deviation. The VIP-200 is placed over the participant’s eye (Figure 2). The Gazept Remote Eye Tracker emits near infrared (NIR) light over the participant and captures participant’s eyes using two optically zoomed NIR cameras. It constantly captures the pupils of the participant reflecting any changes in their size in real-time. For the device under test, a web camera, we use a Microsoft Lifecam [48] of resolution 1280x720 pixels. The web camera stores the video of the participant during each task at 15 frames per second.

Figure 1 -- Experiment Setup

Figure 2 -- Calipers and the VIP-200 in use

Distance between participant’s eyes is measured using a pair of calipers (Figure 2). This measurement is used to convert the pupil size in pixels to size in mm. Prior to performing any task, each participant’s baseline pupillary diameter is determined using the VIP-200 for both eyes separately as well as using a remote eye tracker. The VIP-200 is used to determine the baseline only, veri-fying that the Gazept pupil dilation measurement is accurate to within 0.25 mm. After verification, the VIP-200 is removed from over the participant’s eye.

3.1 Participants In this study we recruited 12 subjects all but 2 were associated with SMU either as a student or staff. At the onset, each partici-pant was asked to fill out a questionnaire on an iPad application about his or her medical history, caffeine intakes, etc. This infor-mation was collected primarily to determine if a participant should be excluded from the experiment. Following Table 1 shows the demographics of the participants.

pupilware 20 participants

over 500 iterations of digit span tasks using both laptop embedded camera and smartphone

for light brown and colored eyes: identical to pupillometer captures sub millimeter pupil response first automatic classification of cognitive load

pupilware 20 participants

over 500 iterations of digit span tasks using both laptop embedded camera and smartphone

for light brown and colored eyes: identical to pupillometer captures sub millimeter pupil response first automatic classification of cognitive load

markers of painPupilWare

head injury

cravings context aware computing

attention

future work in

sympathetic nerve damage fatigue and sleep deprivation

pupillary response

lung functionjaundice

spirometer

device that measures amount of air inhaled and

exhaled.

using a spirometer

flow

volume

volum

e

time

using a spirometer

flow

volume

volum

e

time

flow

volume

volum

e

time

FEV1

FVC

FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity

FEV1% = FEV1/FVC

> 80% healthy60 - 79% mild40 - 59% moderate

< 40% severe

clinical spirometry

flow rate volume

lung functionairflow sensor

sound pressure microphone processing

estimated

SpiroSmart Shwetak PatelMayank Goel

Gaetano Boriello

Jim StoutMargaret Rosenfeld

Elliot Saba

2012 study: appropriate for

trending and screening

if you have access to a smartphone

2012: Using SpiroSmart

2015: Using SpiroCall

SpiroCall

any phone in the world can be used as a spirometer

2015 study: 50 participants

4 styles of phone including feature phone

head to head with spirometer

SpiroCall vs spirometer ~5% spirometer vs spirometer ~5%

in 10 years, COPD will surpass AIDS/HIV as the leading cause of death in low income nations

global initiative for chronic obstructive lung disease (GOLD 2014) world health organization, global burden of disease (2013)

ongoing research third party validation FDA device approval gamification and compliance, reliability scalable training of public health workers

pupillary response

lung functionjaundice

> slide to unlock

Thank You!

eric c. larson | eclarson.com

mobile health for the masses

Assistant Professor Computer Science and Engineering

health monitoring using mobile phonescollaborators: Suku Nair Eric Bing Sohail Rafiqi Mark Wang Ephrem Fernandez, MD, PhD Gaetano Boriello Shwetak Patel Jim Stout, MD Jim Taylor, MD Margaret Rosenfeld, MD Mayank Goel Lilian DeGreef Joseph Camp

eclarson.com eclarson@lyle.smu.edu @ec_larson

eric c. larson | eclarson.comAssistant Professor Computer Science and Engineering

BiliCamhome screening for newborn jaundice

SpiroCallmobile and smartphone spirometry

MobiScreenmobile training for cervical cancer screening

PupilWarepupillary response using

everyday cameras