1 ALMA Lessons Learned for ng-VLA NA ALMA Science Operations: Lessons learned for ngVLA J. E. Hibbard

1 ALMA Lessons Learned for ng-VLA

NA ALMA Science Operations: Lessons learned for

ngVLAJ. E. Hibbard


ALMA Lessons Learned about Science OperationsOr: “What I know now that I wish I had known then”• Biased view after 10 years of detailed ALMA Science

Operations plan development, implementation & support– ALMA Operations Plan (AOP) & 2006 review– 2006 NAASC Plan & review– 2009 ALMA Science Operations Implementation plan & review– 2010 NAASC Plan & reviews– 2014 CONOPS plan & review– 2015 ALMA Operations plans & review

• And 4+ years of ALMA science operations– Cycle 0: Oct 2011 – Dec 2012 (15 mo)– Cycle 1: Jan 2013 – May 2014 (17 mo)– Cycle 2: Jun 2014 – Sep 2015 (16 mo)– Cycle 3: Oct 2015 – Sep 2016 (12mo)


ALMA Science OperationsDetailed Operations Plan was crucial!

• Operations plan developed 2003 – 2005; reviewed 2006 & accepted Feb 2007 – Built upon higher level ALMA Project Plan which

described basic technical specifications, management structure, & high level operations policies & principles

• Final version developed in coordination with all stakeholders & vested experts (technical, management, construction, operations, software, computing…) through series of f2f meetings– Necessarily large. Surprisingly constructive! (But basic

principles agreed to prior to my involvement in 2005)• The 2007 review conclusion: “The Committee felt

that the ALMA Operations Plan is more advanced than for any other projects in a comparable stage.”


ALMA Operations Plan

• Defines:– Fundamental goals– Management structure & responsibilities– High level principles & assumptions– Complete Data flow– Overall workflow– Organization of departments & responsibilities– Staffing levels & methodologies– Budget levels & methodologies


ALMA Operations PlanWhat we got wrong (top level, from SciOps POV)• Put too much into a high level document that required Board

approval for modification– should have clearly delineated high-level concepts/policies

that are controlled by Board from processes that are meant to evolve as experienced gained and not subject to Board approval

– should have delineated long-lasting concepts from those that need yearly approval

• Hiring operations staff to learn side-by-side with construction & commissioning– Requires strong training program and common work

prioritization (not possible with different funding streams?)• That JAO staff could do all other process in the time between

shifts (multitasking is inefficient!)


ALMA Operations PlanKey omissions• Orphaned end-2-end process (gap between a few successful

commission observations and being able to offer a new capability to users)

• Science requirements tracking & interaction w/SW– Later introduced Subsystem scientists, SCIREQ system,

overall process with computing for each observing season• Calibration plan (or process) not detailed enough

– impact on Phase 2 and pipelines– Tension between “just do it” and well defined downstream

processes (metadata, software)• Pipeline developed “on the fly” (should it have been

construction deliverable?)• No plan for documenting procedures & making them available

across the project (document management)• Include professionalism, information flow, authority chain in

“high level principles”


ALMA Operations PlanOverall, detailed AOP was critical for early ALMA success• While document was not perfect, it served us very

well for our implementation and early years of operations

• Importantly, it was generated by a broad consortium of stake holders, represented a broadly accepted model, and formed a basis for common understanding for years to come

• Had enough details on processes & project overview to give staff idea of big picture and where we were going


Lessons learnedHow applicable for ng-VLA?

• How “capability complete” will ng-VLA be when operations takes over?– ALMA has had significant new capabilities in each of the 1st

four cycles• How separate will construction & operations be?

– ALMA “colors of money” and different priorities & processes lead to sub-optimal training & transition of responsibilities

• How distributed will ng-VLA science operations be?– ALMA Science Operations is VERY distributed, with critical

tasks spread over 4 continents & 13 countries– Distributed processes ALWAYS come at higher cost in

staffing, training, communication, process control (but … for ALMA, also gave access to separate funding streams)


Lessons learned: Adopt concept of Standard & non-Standard observations• Introduced in ALMA Cycle 2 (+3yrs)• Standard = demonstrated before they are offered!!!

– These should be understood end-to-end (including pipeline!)

• Non-standard = not as well characterized and/or more work intensive (e.g. manual reduction or complicated Phase2)– Should still be well understood & demonstrated

“adequately”• Time allocated to non-standard observations is

capped • Get this concept adopted at high level so that policy

makers or other committees do not over-ride each cycle!


Lessons learned:Develop software observing simulation system early on• Many tests don’t require on-sky time• Gives developers tool to help with their internal

testing • Would save time for final acceptance testing and

facilitate acceptance process


Lessons learned: Data Reduction Pipeline should be an early deliverable• For ALMA, Calibration PL for standard modes delivered in Cy2

(+3yrs), Imaging for Cy3 (+4yrs)• Must resolve conflict – scientists like to do things “on the fly”;

software needs stable interfaces and data structures and clear intents

• Need detailed and well defined calibration plan: what calibrations go into each mode & how they will be used – ALMA seems to figure these out as we go along

• Adopting standard/non-standard model early would have greatly helped– Initial standard modes should be those we understand

observing, calibration, & reduction path– Make Pipeline work for standard modes – do this from Cycle

0• Currently, PL has a “1 year ahead of time” rule (needs real

ALMA data taken exactly as ALMA plans to execute 1 year ahead of time). Is there a way to get around this (simulated data?)


Immediately Run Calibration Pipeline on individual SB executions (Added post-talk)• SB=schedule block, the atomic unit of an individual Science

Goal from a proposal. Its run repeatedly (often on different days) until the sensitivity requested by the PI is met

• OUS = ObservingUnitSet = set of all executions of an individual SB

• ALMA only runs on complete OUS – so only after all executions of an SB have been obtained. Can be weeks or months after data obtained, so feedback is “stale”

• Running immediately allows you to get immediate feedback on issues with array (see talk by C. Chandler on JVLA)

• Can give PIs early access to calibrated data while project is in-flight (and that’s a good thing!!)

• No processing benefit to doing calibration only after OUS completed


Lessons learned:Take ownership of science requirements early & define process with computing• Requirements originally “owned” by construction; needed to

transition ownership to specific Operations staff (subsystem scientists)

• ALMA is actively evolving system – many capabilities/observing modes still being developed– Important that computing & science operations understand

timelines for providing input, delivering capabilities– Software requirements definition/tracking/testing/acceptance is

significant (un-scoped) effort that only recently go put on regular schedule that is (fairly) well understood across the project

• ALMA developed Subsystem scientist role, “Obsmode” process, SCIREQ tracking system, and requirements/development/delivery timelines “on the fly”. Build them in from the start!


Lessons learned:Implement cohesive system for requirements management• Subsystem Scientists for each software subsystems; Working

groups for each major process– Aided by “cognizant leads” from each Executive– Responsible for requirement tracking, prioritization, testing, “best

practices”

• Separate JIRA System for Science Operations – Allows input from across the project on improved processes, new

features, – Individual “components” managed by Subsystem Scientists and/or

assigned working groups – Keeps unnecessary “chatter” out of SW JIRA (reserved for clear

instructions to developers) – Suggested changes in behavior/science requirements need to be

understood widely and impacts on other subsystems identified

• “Obsmode” group for yearly development cycle associated with each Call– Includes software developers, subsystem scientists & telescope

experts– Sets overall priorities and manages scope & timelines to keep

schedule


Lessons learned: Documentation Centralize key documents & assign upkeep to process owners• Important areas have changed significantly with each cycle

– Data reduction, AoD, Phase 2 preparation, Contact Scientist duties, …

• Access to current procedures is especially important for ALMA, since operations staff are spread across 4 continents and can come and go on short time scales (especially ARC node staff) – less relevant for ng-VLA?

• Science Operations still has not adopted proper documentation system– Chose not to use construction EDM procedure, but did not get

alternative in place– Many documents produced and circulated via JIRA or email, but do

not have proper document numbers, signatures, versions, and usually always listed as “Draft”

• ALMA Procedures described in collection of wiki’s and documents. Not clear these are all kept up to date, or that all group members are working from proper versions


Lessons learned: Documentation (cont’d) Centralize key documents & assign upkeep to process owners• New plan is to have a “Staff Manual” that maintains current

version of all procedures. Simply points to wiki for material that is still dynamic (e.g. operator or AoD procedures)

• Each section (will be) assigned to task leader charged with keeping it up to date

• Will be posted centrally using “internal” area of ALMA Science Portal– Can show different content based on different user “roles”, so

using this to have an internal area for world-wide

• (Still waiting on process for documentation control & acceptance process)

• Additional lesson: include document management in planning?


• ALMA Staff Manual


Lessons learned:Define Roles & Access control

• Software tools provide wide access to sensitive data• As mentioned, ALMA Science Operations has staff

spread over 4 continents, often with short turn-over• Currently trying to reverse-engineer data access

control• Would have been much better to define roles and

access control much earlier in process (in first implementation plan). E.g.– Contact scientists – limited and read-only access, only

to projects assigned to support– Helpdesk staff – broad read-only access– Phase 2 experts – limited and modify access – AoD – broad modify access– Etc. (data reducers, data reduction managers, archive

managers …)


Lessons learnedUser Support• Biggest complaint is lack of information

– Specify transparency as a high-level principle

• ALMA partners have different interpretations about SRDP– No access *until* data are ready!– Feeling that there is a need to protect users from the data

• Tools– Observing tool, sensitivity calculator, helpdesk, documents all get

good marks– Project Tracker doesn’t present information users are most

interested in (being re-implemented)– Archive search needs improvement (staff aren’t using it!)

• One-on-one support– Helpdesk: commercial solution does not lend itself to re-use of

ticket resolutions• Combine helpdesk with “forums”?

– F2f support: we built it, but they did not come (yet)


Lessons learnedUser Support Model• ALMA distributes user support to ARC/ARC node system• Recent Operations Review (May 2015) recognized this as one

of the successes & strengths of ALMA operations


Lessons learnedUser Support Model (cont’d)• Benefits:

– Each region has 25-50 scientific staff who connect to distributed user base

– Core ALMA only pays for a fraction of these (leverage regional interest to get alternative funding streams)

– Provided incredible capability to compensate for unforeseen issues (e.g. manual pipeline, ph2 process, requirements tracking) – comes at expense of key mission

– Provides large base with broad expertise that could not be recruited to Chile

• Drawbacks:– Divorced from expertise & realities at telescope– Communication & keeping current– Training – Persistency– Cost


Lessons learned:Science Ready Data Products (SRDP)• ALMA delivers calibrated data and imaging products (but often

only for subset of observations)– Done through a very work-intensive heterogeneous process (but

that is changing)

• However, this adds months onto timescale for user’s access to data – No access until data are ready!

• It is clear that some PIs don’t want the products anyway! Visibility plane analysis (proto-planetary disks; gravitational lens modeling)

• Still, ALMA has succeeded in reaching out to non-traditional optical/IR community, and they are excited

• My recommendation:– It’s a mistake to raise the bar so high that it overly delays

delivery to PIs – start low (best efforts), and build– But not an excuse to deny access to PIs– I would build-in multi-access – raw data, calibrated data,

image products


Lessons Learned from ALMA Science OperationsSummary

• Detailed operations plan is critical• Early hiring of operations staff: either have proper

training in place, or don’t bother• Clarify responsibility for end-to-end process• If capabilities evolve, use concept of standard/non-

standard• Make pipeline an early (construction?) deliverable• Pay early attention to some of the details

– Science/software interaction– Document management– Centralized procedures– Access control


www.nrao.eduscience.nrao.edu

The National Radio Astronomy Observatory is a facility of the National Science Foundation

operated under cooperative agreement by Associated Universities, Inc.


Lessons learned: Keep things going! Avoid episodic panic mode• ALMA had periodic reviews + several instances of reviews

being planned but canceled. “Hurry up and wait”• Science Operations worked out details of many processes for

Implementation Plan Review in 2009– Really helped us solidify procedural details– However, while processes were implemented, process

documents often fell to the side– Lead to panic mode for subsequent reviews– Often surprised by how good earlier documents were when

we went to update them!– Could have saved time & effort if these key documents

were assigned to task leaders to keep up to date as processes were implemented & evolved

– Paid off at last Operations Review (SciOps had completed their work for the defunct AOPvE effort; updating for review was relatively straight forward)


Lessons not learned:Keep science drivers forefront

• As decisions are made, ALMA has risk of “random walk” away from top-level science drivers that lead to its strong support in the community– Number of array elements available– Observing efficiency


Lessons not learned:Complicated & Expensive Panel review

• ALMA conducts a face-to-face panel review system• Complicated to arrange and very expensive• Does the final result justify the cost & complication?


Lessons not learned:Let scheduling committee decide final outcome!• ALMA lets panel ranking decide grades• Resulting proposal pressure leads to uncompletable

observing queue

LST

Too many projects

Too few projects


Lessons not learned:Is there a way to avoid significant dead-end efforts?• CONOPS effort: define “concept of operations” that

shows that budget decisions can be driven based on overall performance of the array

• Requested by Board; organized by Director• ~6mo effort from 3 staff• Participation of 21 domain experts & key staff• Generated 130 pg report, internally reviewed• Nothing changed


Documents

1 ALMA Lessons Learned for ng-VLA NA ALMA Science Operations: Lessons learned for ngVLA J. E. Hibbard