Yelp Tech Talks: Mobile Testing 1, 2, 3

@YelpEngineering

YelpEngineers

engineeringblog.yelp.com

github.com/yelpyelp.com/careers

Building Yelp for Apple WatchBill Meltsner

wmeltsner@yelp.com@billmeltsner

Today

Initial Scoping / Planning

Yelp.app on Apple Watch Deep Dive

Lessons Learned

Who I Am

iOS Technical Lead

Yelp on Apple Watch project lead

Worked largely on Watch app logic

Who We Were

Designer Engineer Product Manager Engineer (Intern)

Why Build Yelp Watch App?

Yelp everywhere

Day-one advantage

Knew we could build a great experience

Initial Scoping

time = features

Initial Scoping

unlimited time* = unlimited features

*this never happens

Initial Scoping

finite time = finite features

Initial Scoping

Product team defined features

Engineering team estimated available time

Worked together to define MVP as cost of time versus feature

Initial Scoping

UI & Logic could be parallelized- 1 engineer for each

Milestones for MVP, MVP+1, MVP+2, …

Flexibility in our schedule to add / remove features and stay agile

Demo

Yelp Watch AppTechnical Overview

WatchKit - A Changing Landscape

Brand new platform with docs & APIs changing drastically between betas

No defined best practices

Focused on making best-effort technical decisions with the ability to refine later

Overview of a WatchKit app

Parent App

API Requests

Watch App

Storyboard

WatchKit Extension

Location

Logic

Interface Control

Images

iPhone Apple Watch

Interface Controllers

View Controller analog

Interface hierarchy is fixed

YPWKSearchResultsInterfaceController

Storyboard Overview

Before After

Networking

API requests owned by parent app

Image loading owned by extension

Location

Owned by extension – runs in foreground, parent app runs in background

We only request foreground access

Location

Permissions belong to parent app, must be granted on phone

Phone ↔ Watch Communications

All calls in one iteration of the run loop coalesced together

Communication between watch and phone is rate-limited serial queue

Overhead is high – batch your calls!

Images

Key part of our search UI

Naive approach: send each image to the watch as it’s loaded

Result: traffic jam of communications, unresponsive app

Images

Solution: Wait x seconds, send all images loaded in that timeframe at once

Problem: that can be a lot of data

Solution Part 2: crush the heck out of ‘emUIImageJPEGRepresentation(image, 0.0) // max

compression

Lessons Learned

Think like a Startup

Priority 1 was being there on launch day

MVP comes first, everything else can wait

Technical debt is not inherently bad

Plan Ahead

Define designs and scope before writing any code

New platforms are hard to predict effectively

Bend but don’t break

Questions?

Testing The Yelp AppiOS & Android

Who We Are

Mason Glidden● iOS Engineer● iOS Testing

o KIF & Jenkins● mglidden@yelp.com

Tim Mellor● Android Engineer● Android Testing

o Espresso & Jenkins

How we develop new mobile APIs

iOS & Android - Tests & Testing Strategy

Today

Building New Mobile APIsHow we use our documentation to test new APIs

Mobile APIs @ Yelp

● API shared by iOS & Android● New APIs start with documentation and

examples● Client and API can be developed simultaneously● API team manages backwards compatibility tests

/*h2. Photo (full)

|_. Name |_. Type |_. Description || id | string | Identifier || time_created | time | Timestamp for when photo was uploaded || url_prefix | string | Prefix for image url || user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) || caption | string | Caption || feedback_positive_count | integer | Number of likes for the photo ||*/{ "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3-media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19,}

Documentation

/*h2. Photo (full)

|_. Name |_. Type |_. Description || id | string | Identifier || time_created | time | Timestamp for when photo was uploaded || url_prefix | string | Prefix for image url || user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) || caption | string | Caption || feedback_positive_count | integer | Number of likes for the photo ||*/{ "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3-media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19,}

Documentation

Textile

/*h2. Photo (full)

|_. Name |_. Type |_. Description || id | string | Identifier || time_created | time | Timestamp for when photo was uploaded || url_prefix | string | Prefix for image url || user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) || caption | string | Caption || feedback_positive_count | integer | Number of likes for the photo ||*/{ "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3-media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19,}

Documentation

JSONDoc

Textile

Documentation/*h2. Photo (full)

|_. Name |_. Type |_. Description || id | string | Identifier || time_created | time | Timestamp for when photo was uploaded || url_prefix | string | Prefix for image url || user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) || caption | string | Caption || feedback_positive_count | integer | Number of likes for the photo ||*/{ "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3-media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19,}

Documentation -> JSON

● Included as submodule in client repos● Build step to flatten documentation into JSON

(e.g. v1--objects--photo+full.json)● Code requests specific mocks

Why This Approach Works for Us

● API & client contract● Fewer dependencies for developers & Jenkins● Improved test speed & reliability on iOS &

Android

iOS Testing @ Yelp Mason Glidden

● Prevent Regressions● Give developers confidence● Run quickly● Reliable results● Easy to write

Test Goals

● Unit Tests● Integration Tests● Acceptance Tests

Test Types

● Prevent UI & Logic Regressions● ~150 logic unit tests● ~100 network request contract

tests● ~650 view tests● Continuous Integration on Jenkins

Unit Tests

● Generally pretty simple● Test-Driven Development● Super fast to run

Logic Tests

Example: Business Hours Logic- (void)testOpensSoon { // Test that "Opens soon" appears with the correct time interval [NSDate yk_setDate:[NSDate dateWithTimeIntervalSince1970:1356040364]]; NSArray *openHoursArray = @[@[@5160, @5460]]; OpenHours *openHours = [OpenHours openHoursFromJSON:openHoursArray timeZoneString:@"America/Los_Angeles"]; STAssertEqualObjects(@"Opens in 8 min", [openHours openOrClosedStringUsingMinutes:YES], nil);}

Logic Tests

● Makes sure client can still parse documented API changes

● Example: ReviewsListRequestTest- (void)testList { ReviewsListRequest *request = [[ReviewsListRequest alloc] init]; [OHHTTPStubs yp_receiveFromPath:@"v1--reviews+reviews.json" statusCode:200 MIMEType:@"application/json" afterDelay:0.1]; [request listWithBusinessId:@"BIZID" selectedReviewId:nil offset:0 limit:10 delegate:self]; [self waitForStatus:YPAsyncTestWaitStatusSuccess timeout:10.0 requestToCancelOnTimeout:request];}

Parsing Tests

View Tests

● View with mock data

● Screenshot of view● Compares with

previous versions● Based off GHUnit

View Tests

View Tests

● Example: contribution buttons view test- (void)testBasicButtons { Business *business = [Business businessFromJSONDictionary: [YPDebug JSONFromResource:@"v1--objects--business+full.json"] request:nil context:nil]; YPBusinessContributeButtons *buttons = [[YPBusinessContributeButtons alloc] init]; [buttons setBusiness:business]; YPVerifyView(buttons);}

View Tests

● Pros:○ Easy way to catch regressions○ Invaluable when refactoring or updating to

new OS versions● Cons:

○ Slow: ~¾ seconds per test○ Lots of false-positive failures

Integration Tests

● Testing that application behaves as expected● Interaction between view controllers● Primary signals of a problem:

○ Non-visual - analytics & network requests○ Visual - button or label

● ~225 Integration tests

KIF

● ~150 KIF tests● Uses accessibility labels to navigate● Custom hooks for analytics,

requests● Continuous integration on Jenkins● Separate iPad and iPhone tests

github.com/kif-framework/KIF

Integration Test Example

Integration Test Example● Example: ReviewCompositionIntegrationTest

- (void)testReviewWrite { [YPAPI yp_addMockSessionConfirmed:YES sendNotification:YES]; [self yp_openBusinessViewController];

[tester tapViewWithAccessibilityLabel:@"Write a Review"]; [tester yp_waitForModalViewControllerOfClass:[ReviewComposeViewController class]]; [tester yp_waitForExpectedAnalytics:@[@{@"iri": kAnalyticsReviewWriteIRI, @"params": @{@"intended_compose_type": @"add"}]];

[tester tapViewWithAccessibilityLabel:@"Rating"]; [tester clearTextFromAndThenEnterText:@"My review" intoViewWithAccessibilityLabel:@"Write your review here."]; [tester tapViewWithAccessibilityLabel:@"Post"]; [tester yp_waitForRequestWithPath:@"/review/save" queryParams:nil postParams:@{ @"text": @"My review", @"rating": @"3" }];}

Integration Test Example● Example: ReviewCompositionIntegrationTest

- (void)testReviewWrite { [YPAPI yp_addMockSessionConfirmed:YES sendNotification:YES]; [self yp_openBusinessViewController];

[tester tapViewWithAccessibilityLabel:@"Write a Review"]; [tester yp_waitForModalViewControllerOfClass:[ReviewComposeViewController class]]; [tester yp_waitForExpectedAnalytics:@[@{@"iri": kAnalyticsReviewWriteIRI, @"params": @{@"intended_compose_type": @"add"}]];

[tester tapViewWithAccessibilityLabel:@"Rating"]; [tester clearTextFromAndThenEnterText:@"My review" intoViewWithAccessibilityLabel:@"Write your review here."]; [tester tapViewWithAccessibilityLabel:@"Post"]; [tester yp_waitForRequestWithPath:@"/review/save" queryParams:nil postParams:@{ @"text": @"My review", @"rating": @"3" }];}

Integration Test Example● Example: ReviewCompositionIntegrationTest

- (void)testReviewWrite { [YPAPI yp_addMockSessionConfirmed:YES sendNotification:YES]; [self yp_openBusinessViewController];

[tester tapViewWithAccessibilityLabel:@"Write a Review"]; [tester yp_waitForModalViewControllerOfClass:[ReviewComposeViewController class]]; [tester yp_waitForExpectedAnalytics:@[@{@"iri": kAnalyticsReviewWriteIRI, @"params": @{@"intended_compose_type": @"add"}]];

[tester tapViewWithAccessibilityLabel:@"Rating"]; [tester clearTextFromAndThenEnterText:@"My review" intoViewWithAccessibilityLabel:@"Write your review here."]; [tester tapViewWithAccessibilityLabel:@"Post"]; [tester yp_waitForRequestWithPath:@"/review/save" queryParams:nil postParams:@{ @"text": @"My review", @"rating": @"3" }];}

Integration Test Example● Example: ReviewCompositionIntegrationTest

- (void)testReviewWrite { [YPAPI yp_addMockSessionConfirmed:YES sendNotification:YES]; [self yp_openBusinessViewController];

[tester tapViewWithAccessibilityLabel:@"Write a Review"]; [tester yp_waitForModalViewControllerOfClass:[ReviewComposeViewController class]]; [tester yp_waitForExpectedAnalytics:@[@{@"iri": kAnalyticsReviewWriteIRI, @"params": @{@"intended_compose_type": @"add"}]];

[tester tapViewWithAccessibilityLabel:@"Rating"]; [tester clearTextFromAndThenEnterText:@"My review" intoViewWithAccessibilityLabel:@"Write your review here."]; [tester tapViewWithAccessibilityLabel:@"Post"]; [tester yp_waitForRequestWithPath:@"/review/save" queryParams:nil postParams:@{ @"text": @"My review", @"rating": @"3" }];}

Sandboxing

Mocked During Tests:● Networking● Date, Time, Timezone● Device permissions● Singletons

Between Test Runs:● Clean caches● Reset user defaults● Reset navigation stack● Device orientation

Other Tooling

● OHHTTPStubs to block & mock network requests

● OCMock for mocking● XCTool to run our tests

Acceptance Tests

● Test overall look and feel● ~50 manual test cases

○ Moving some to KIF● iOS7 & 8, iPad & iPhone● Run by Engineers + PM during release

process

Closing Thoughts

● API mocks make it easy for us to reliably grow our testing suite

● Different types of tests for different problems● Sandboxes to create consistent environments● KIF <3

Android Testing @ YelpTim Mellor

tfmellor@yelp.com

tests = tools + code

Simple, right?

● AndroidTestCase● InstrumentationTestCase● ApplicationTestCase● ActivityTestCase● ActivityUnitTestCase● ActivityInstrumentationTestCase● ActivityInstrumentationTestCase2

More decisions

Problem: Devices

● Devices are necessary

● Devices suck● Virtual devices

are bearable

Solution: Devices

● Genymotion’s gmtool● Clone image into new device● Speed of Genymotion

$ python gmtool_wrapper.py start \ --vms '{"18":1, "19":1, "21":1}'

Problem: Flakes!

● Part 1: Instrumentation + Device● Part 2: Test library

Android Instrumentation

Instrumentation consequences

● Uncaught exceptions halt test suite● Activities/Services/etc. stay open

Solution: Instrumentation Flakes

● One test per instrumentation run!$ adb shell pm clear com.yelp.droid

Flakiness and Test libraries

Robotium and its solo.waitFor* methods

Android test kit to the rescue!

Main Thread

click()

Espresso

blocked

Main Thread

Test thread

assertionsTest

click()

Espresso

blocked

Main Thread

Test thread

assertionsTest

Task ThreadBackground task

Problem: slow test suites

● Consequence of needing devices● Long = impractical

Solution: test sharding

$ adb shell am instrument -w \ -e numShards 4 \ -e shardIndex 1● github.com/shazam/fork● Resources are the limit!

Yelp Testing Process

● ~300 unit tests● ~100 integration tests● ~150 UI integration tests● Manual testing against production● Beta group● 50% roll-out in Play Store

UI Integration test toolkit @ Yelp

● Espresso!● Home-rolled MockHttpClient

o MockResponseo MockRequestMatcher

● Spoon

public void test_ClickingBookmark_SendsAddRequestAndUpdatesUi() { mBusiness.setBookmarked(false); setActivityIntent(ActivityBusinessPage.intentForBusiness(getYelpContext(), mBusiness)); getActivity(); takeScreenshot("business detail unbookmarked");

// Check that the bookmark button has the text “Bookmark” and click on it // This should trigger a bookmark add request. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.action_bookmark)))) .perform(click()); takeScreenshot("bookmarked");

// Make sure we hit bookmarks/add with the proper params. assertThat(mAddBookmarkResponse.getRequestCount(), is(1)); // Make sure the "Bookmarked" button now shows. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.bookmarked))));}

public void test_ClickingBookmark_SendsAddRequestAndUpdatesUi() { mBusiness.setBookmarked(false); setActivityIntent(ActivityBusinessPage.intentForBusiness(getYelpContext(), mBusiness)); getActivity(); takeScreenshot("business detail unbookmarked");

// Check that the bookmark button has the text “Bookmark” and click on it // This should trigger a bookmark add request. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.action_bookmark)))) .perform(click()); takeScreenshot("bookmarked");

// Make sure we hit bookmarks/add with the proper params. assertThat(mAddBookmarkResponse.getRequestCount(), is(1)); // Make sure the "Bookmarked" button now shows. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.bookmarked))));}

public void test_ClickingBookmark_SendsAddRequestAndUpdatesUi() { mBusiness.setBookmarked(false); setActivityIntent(ActivityBusinessPage.intentForBusiness(getYelpContext(), mBusiness)); getActivity(); takeScreenshot("business detail unbookmarked");

// Check that the bookmark button has the text “Bookmark” and click on it // This should trigger a bookmark add request. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.action_bookmark)))) .perform(click()); takeScreenshot("bookmarked");

// Make sure we hit bookmarks/add with the proper params. assertThat(mAddBookmarkResponse.getRequestCount(), is(1)); // Make sure the "Bookmarked" button now shows. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.bookmarked))));}

public void test_ClickingBookmark_SendsAddRequestAndUpdatesUi() { mBusiness.setBookmarked(false); setActivityIntent(ActivityBusinessPage.intentForBusiness(getYelpContext(), mBusiness)); getActivity(); takeScreenshot("business detail unbookmarked");

// Check that the bookmark button has the text “Bookmark” and click on it // This should trigger a bookmark add request. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.action_bookmark)))) .perform(click()); takeScreenshot("bookmarked");

// Make sure we hit bookmarks/add with the proper params. assertThat(mAddBookmarkResponse.getRequestCount(), is(1)); // Make sure the "Bookmarked" button now shows. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.bookmarked))));}

public void test_WriteTip_SendsProperRequest() { setMockSession(); MockResponse tipSaveResponse = mMockHttpClient.addDefaultMock( ApiPath.QUICKTIPS_SAVE, getTipSaveParams());

openTipPage();

onView(withId(R.id.edit_text)).perform(typeText(TIP_TEXT)); takeScreenshot("tip typed");

onView(withId(R.id.done_button)).perform(click()); takeScreenshot("business page");

// We should show a "Thanks for the tip!" dialog on the business page. onView(withText(R.string.thanks_for_the_tip)).check(matches(isDisplayed())); assertThat(tipSaveResponse.getRequestCount(), is(1));}

public void test_WriteTip_SendsProperRequest() { setMockSession(); MockResponse tipSaveResponse = mMockHttpClient.addDefaultMock( ApiPath.QUICKTIPS_SAVE, getTipSaveParams());

openTipPage();

onView(withId(R.id.edit_text)).perform(typeText(TIP_TEXT)); takeScreenshot("tip typed");

onView(withId(R.id.done_button)).perform(click()); takeScreenshot("business page");

// We should show a "Thanks for the tip!" dialog on the business page. onView(withText(R.string.thanks_for_the_tip)).check(matches(isDisplayed())); assertThat(tipSaveResponse.getRequestCount(), is(1));}

public void test_WriteTip_SendsProperRequest() { setMockSession(); MockResponse tipSaveResponse = mMockHttpClient.addDefaultMock( ApiPath.QUICKTIPS_SAVE, getTipSaveParams());

openTipPage();

onView(withId(R.id.edit_text)).perform(typeText(TIP_TEXT)); takeScreenshot("tip typed");

onView(withId(R.id.done_button)).perform(click()); takeScreenshot("business page");

// We should show a "Thanks for the tip!" dialog on the business page. onView(withText(R.string.thanks_for_the_tip)).check(matches(isDisplayed())); assertThat(tipSaveResponse.getRequestCount(), is(1));}

public void test_WriteTip_SendsProperRequest() { setMockSession(); MockResponse tipSaveResponse = mMockHttpClient.addDefaultMock( ApiPath.QUICKTIPS_SAVE, getTipSaveParams());

openTipPage();

onView(withId(R.id.edit_text)).perform(typeText(TIP_TEXT)); takeScreenshot("tip typed");

onView(withId(R.id.done_button)).perform(click()); takeScreenshot("business page");

// We should show a "Thanks for the tip!" dialog on the business page. onView(withText(R.string.thanks_for_the_tip)).check(matches(isDisplayed())); assertThat(tipSaveResponse.getRequestCount(), is(1));}

● Library choices matter● Address the issues at the source!● Tests don’t have to be a pain

Lessons learned

Questions?