BioData Development Tech Talk

• Project Structure• Application Features • Technology Stack• Development Methodology/Practices • Tooling

Project Overview/Objectives

• A secure, modern, web-oriented, extensible and adaptable system for the capture, storage, curation, and distribution of bioassessment and supporting habitat data commonly collected for NAWQA and other USGS projects.

• Version 1 to support stream macroinvertebrate, algal periphyton, fish-community and instream habitat data collected using protocols of USGS National Water-Quality Assessment Program (NAWQA) and the USEPA National Rivers and Streams Assessment (NRSA).

Goal

In this meeting the Development Lead (Mike Stern) for the Biodata project, will discuss several key areas of the project, and what he views as keys that led to a successful project. Discussion and questions are encouraged.

About Me• Work for Comprehensive Computer Consulting (CCC) out of Sun Prairie, WI • Work out of Wisconsin Science Center…co-working with CIDA in Middleton, WI• Java developer for 12 years, mostly enterprise level in-house applications• Development lead on BioData—prefer server side/persistence, web api• Love doing code reviews, and refactoring

Project Structure• Clients: Project managers/Business analysts • Development team: (Small, never > 8 devs at one time)

– CCC devs are prime (3-5 active developers)– USGS devs rotate in and out, (1-4 active developers)– CIDA administration/hosting services (OS, Oracle, network, desktop &

infrastructure) and coworking facility• Agile--Both Client and Developer buy in 2 week iterations. 3-4

meetings per iteration: planning, mid iteration, preplanning, quarterly planning.

• Daily standups (separate client and developers) • 3 main projects (Data Entry, ETL, Reporting/warehousing) • Integrations with Legacy Inputs (NWIS Sites, BioTDB, Labs)

Clients

• Adopted the Lean/Agile methodology into their processes

• Story and Acceptance Test Driven Development• Confluence (WIKI) and JIRA (task management

system) are main communication tools.• Understand the iterative development cycle

and have adopted Agile practices within their group

Developers

• Small Team (< 8 devs at all times usually around 4 active).

• Easy communication, IM, standups, low protocol• Let team self assign. Strengths quickly become evident

& let members gravitate towards them (benefits project and developer alike)

• Constant design discussions, little diagraming aside from whiteboards. Let code speak rather than outdated documents

• Low formality code review

Agile / Lean Software Development Lean Software Development – Mary & Tom

Poppendieck Lean Principals

Eliminate Waste Build Quality In Create Knowledge Defer Commitment Deliver Fast Respect People Optimize the Whole

Daily standups

• About 15 minutes max… 1-2 minute individual reports• What did you do yesterday, what are you doing later

today. Any blockers?• Encourages cross communication and global awareness

of changes in the system (key to integration)• Table lengthy discussions to be discussed immediately

after standup.• Separate meetings for development and business

analyst staff members.

Project Components

• 3 separate ‘Projects’• Biodata Input—Data entry, auto save, allow

‘invalid data’, continuous validation• Biodata Retreival — Canned reports with

flexible (and reusable) query criteria• Biodata ETL — Transforms Biodata input and

Legacy data into a more normalized format for use in retrieval…inject visibility rules

• Quick Demo

Application Features

• REST services to JS front end...clear decoupling • Auto Save with Real time validation (avoid save

buttons) • Top Down/drill down paradigm. Start general

work narrow: Project List of Field Samples Filter to narrow Edit/View Sample

• Quick Search using index tables for ‘random’ access

DEMO

• Biodata Input (QA): https://igsarmewfsbd3b.er.usgs.gov:8443/biodataWeb/

• Biodata Retreival (QA) https://igsarmewfsbd3b.er.usgs.gov:8443/bioshareWeb/

REST: Quick Explanation

Based solely on HTTP – the lingua franca of the Internet

Everything is implemented as a PUT, POST, GET, or DELETE

Everything is a RESOURCE, and everything has a URI

• Dynamic object hierarchy inflation (with api) – https://igsarmewfsbd3b.er.usgs.gov:8443/biodata

Web/service/project/11528– https://igsarmewfsbd3b.er.usgs.gov:8443/biodata

Web/service/project/11528?inflateOption=:(projectOwner,projectStaff)

Technology Stack

• layered java, REST-ful webservice, extJS UI, Oracle

• JBoss AS 5.1, Apache, centOS • Key Java libraries: Spring, Ibatis 2, RESTeasy,

Jasper Reports • Oracle XML tables using hierarchical queries,

prepared statements

System Structure

Reference Application Architecture

DevelopmentSystem

Development Methodology/Practices

• Test Driven Development--Make it Break, Make it Work, Make it Pretty

• Continuous integration: good coverage, encourage early and often check ins

• Push Button Deploy and 'releases' • Light weight process (code reviews, build and

deploy, documentation) • Optimize when necessary

Testing Unit testing – All significant custom code has unit tests using

the Junit test framework. We don't write unit tests for third-party java frameworks, just for the code we write. Encourage Test Driven Development - Tests are written before the

implementing code…helps work out the API & design Tests are the ultimate place to start when onboarding new devs…

can exercise the stack starting with tests…debug through Require tests for custom code check-in. Cobertura used to report

on test coverage. Not religious about coverage, but it’s a good data point.

Developers encouraged to run tests locally, continuous build and integration runs all tests each time code is checked into SCM.

Continuous Build and Integration Jenkins (formerly Hudson) used to implement continuous

build, test, integration, on-demand deploy. Entire system is built after each developer checks in code.

Developers are encouraged to check code in multiple times a day to reduce merge and integration problems.

Unit and integration tests are run each time changes to SCM are detected. May include one or more check-ins.

If build “breaks” entire team is informed. Members may check Hudson “dashboard” to determine who needs to do what.

Release plugin encourages process for code elevation and following the Trunk, snapshot, branches workflow SVN loves so much. Also reinforces maven type development snapshot, Release, and Hotfix mentality.

Tooling:

Rely on automation to make the developers job 'easy'. Leverage open source tools. • Confluence• JIRA• Eclipse • SVN • Maven • Jenkins• Jboss• IReport • Selenium • Remedy