1

Allocations RAT Recommendations

Presentation to TG ARCH meetingJuly 10, 2008

Significant modifications from TG’08 slides are highlighted in red.

Richard Moore

Context

• RAT formed March 2008, subsequent to March LRAC & discussion at March TG Quarterly

• RAT’s charter, meeting minutes and documentation on wiki http://www.teragridforum.org/mediawiki/index.php?title=RATs#Allocations_.282008.29

• Feedback on preliminary recommendations solicited at TG’08 (SAB, MRAC, BOF) & in open survey (57 responses)

• RAT is nearing completion of its deliverables: – Set of recommendations (internal use)– A charter from the TG Forum for the TRAC –Revised “GPG” for resource requests (broad distribution)– Report on results of User Survey –A set of action items to implement the recommendations

• Want to finalize and ratify these documents 2

3

Goals

• Reduce the real and perceived barriers to entry for the allocations process

• Resolve and clarify various policies related to the integration and presentation of TeraGrid resources

• Improve communications about resources and allocations

4

Lowering Barriers to Entry:New Award Types & Review Schedule

• There will be three types of awards: Startup, Education & Research

– Formerly, DAC, MRAC and LRAC awards which reflected size of award

• Startup & Education Awards may be submitted at any time

– Streamline directions and submittal for these “entry point” requests

– TeraGrid team members review and act on these requests as quickly as possible .

– Single TG team reviewer can approve; initial rejection invokes a second review.

– Provide much larger Startup awards for larger systems (default 200K rather than 30K SU’s)

• Startup size can be limited by the Resource Providers for smaller systems

• All Research requests may be submitted to any quarterly review meeting

– No longer a distinction between “Medium” & “Large” requests

– “Large” requests used to be considered only semi-annually

• Change name to TeraGrid Resource Allocation Committee (TRAC)

• TeraGrid will take steps to ensure that larger requests do not preclude the ability of more modestly sized requests to accomplish their research objectives, e.g.

– If resources are over-subscribed, reviewers re-assess large allocations first

– Group the requests into comparable classes during review process

5

Lowering Barriers to Entry: Getting in the door

• Improve communication about resources and process to new users/communities

• NSF will be updating the eligibility requirements for who can request resources

• Provide training for resource requests: TeraGrid advertise and host a 1-2 hour teleconference training & Q&A session for all prospective proposers early in each proposal cycle.

– Provide information about resources available, proposal requirements, tips for writing good proposals, info about start-up accounts for scaling studies in support of proposals.

• PIs – both current and new – can request Advances on future proposal requests

– Only applies to resources with current availability – advances can not crowd out previously allocated users.

– Increase limit on Advances from 10% to 25%, subject to resource avail & RP approval

– If resources are available, Allocations Coordinator and RP’s can grant award or choose to ask TRAC members to review the proposal.

• Maintain current procedures for supplemental requests for existing PI’s– This is significant mechanism in circumstances where resources are not fully allocated

6

Broaden TRAC membership selection process and clarify term limits

• Solicit nominations for TRAC reviewers in the required domain areas from:

– NSF OCI program officers

– Program officers in NSF domain directorates or other agencies (via OCI)

– TeraGrid Science Advisory Board

– Departing TRAC members

– TeraGrid Forum (TG RP’s and GIG)

• Consider breadth of geographical/institutional participation, as well as diversity

• Candidates need not be TRAC awardees, or even eligible for a TRAC award

• Nominees must be approved by the TeraGrid Forum

• Term limit is 3 years, but reviewers can agree to serve for only 2 years

– In exceptional circumstances, Allocations Coordinator can request a 6 month extension

– Previous members can rejoin after a minimum 2-year hiatus

• TRAC members should expect to attend at least 3 or 4 reviews per year (only 3 if they are also a PI/Co-PI – see next page)

• TRAC chair is a committee member who participated in at least the 2 prior meetings; will be nominated/elected by the committee at conclusion of previous meeting. The Chair has no special decision-making privileges.

7

Strengthen Perception of Fairness• Adopt formal conflict of interest (COI) policy, provided to all prospective

candidate reviewers, and explained to TRAC participants at each meeting– Reviewer affiliations with a requestor’s institution; personal, academic and/or financial interest in

the request and/or proposal; other potential COI relationships

• PI can name individuals who they request not review their resource request (comparable to NSF policy). The allocations coordinator will consider such requests. 

• If TRAC reviewers have significantly different opinions on a resource requests, they should be openly discussed in full TRAC meeting.

• A TRAC reviewer who is also a PI or co-PI on a resource request should not review proposals or attend the meeting in the cycle with their request

– (No longer applicable if above is established) There will be no informal discussion among reviewers of any proposal for which the PI or co-PI is a TRAC panel member and is present at the meeting. That PI or co-PI must, of course, leave the room during discussion of his/her proposal.

• TG Forum will develop charter for the TRAC and is responsible for management the allocations process

Clarify meeting participation by TG Team Members

• Attendance by –One designated “Allocations Officer” per RP –GIG: PI or delegate (e.g. Dave); TG Allocations Coordinator (Kent),

Allocations staff member (e.g. Vicki), Logistics Coordinator (Valerie)–No more than one additional representative per RP (may be zero)

• All TG Team Members are observers and should only provide info requested by the committee (enforced by Chair)

• All TG team members except Allocations Coordinator (Kent) will sit in a gallery or side chairs (i.e. “not at the table”)

• All members must maintain confidentiality of discussions.• RP Allocation Officers and GIG representatives would be

exempt from COI provisions (unless they are a PI or co-PI). Any additional RP representatives would be subject to same COI provisions as reviewers. 8

Resource request structure• Main Proposal (required, 10 or 15 pages). This document should have the scientific background, research objectives,

and the justification of the resource request. Within the main document, some flexibility can be permitted. However, a recommended sample template can be provided and/or recommended sections outlined. The reviewers will focus on this document. Most proposals (new, renewals, supplements, justifications) have a 10-page limit. Only multi-year proposals and single-year proposals requesting more than 10 million SUs can use up to 15 pages.

• Progress Report (required for renewals and supplements, 5-page limit). This document should report on progress made during prior (current) allocation award period. This is the main document for Progress Reports within Multi-Year Awards. The reviewers will consider this document for rating the successful use of prior allocation awards.

• Publications Resulting from TeraGrid Support (required for renewals/progress reports of all kinds, including renewed “small” allocations, no page limit; N/A for new submissions). This document should contain a list of bibliographic citations for publications in preparation, submitted, accepted, or published that benefitted from access to TeraGrid resources. Where implementation allows, bibliographic document formats (such as EndNote or BibTeX) could be supported for this attachment. This document is separated for TeraGrid metrics purposes.

• Special Requirements (optional, 1-page limit). This document should be included if the completion of a proposed research plan requires capabilities that fall outside an RP’s standard user and usage environment. For example, jobs longer than the standard queue lengths, specific scheduling demands, or dedicated file system space. The TRAC is not required to read this document; the intended audience is the staff at the relevant RP(s).

• Performance and Scaling of Codes (optional, 5-page limit). This document should contain information related to code performance and scaling, if this information does not fit within the Main Proposal document. This document should be used only to support claims made and used in the Main Proposal regarding the performance of relevant codes. Reviewers will not be required to closely scrutinize this document to evaluate the computational justification.

• References and Figures (optional, no limit).• CVs (required for most submissions, perhaps optional for supplements and justifications). This is a single document that

includes all submitted CVs in NSF or NIH two-page formats.

9

10

Oversubscription of Resources (1 of 2)

• If resources are oversubscribed at the conclusion of the initial TRAC recommendations, the attending RP and GIG allocation officers will formally convene to assess whether alternate resources are available to satisfy total allocations.

• If so, the TRAC committee adjourns and the RP allocations officers will make re-allocations based on, in priority order:

– Re-assigning allocations to alternate resources specified by user in their proposal,

– Re-assigning allocations to alternate resources with similar system architectures,

– Retaining current users rather than assigning new users to oversubscribed resources, &

– Allocating to sites where the user has the most previous experience.

11

Oversubscription of Resources (2 of 2)

• If alternate resources can not satisfy total allocations, further cuts must be made. In that case, the TRAC will reconvene after the TG allocations officers make a best initial pass at re-allocations.

• The TRAC will be asked to identify the least meritorious proposals overall whose awards received a disproportionate level of SUs, with a charge of reducing such allocations until the total recommended awards can be satisfied by available resources.

• Users assigned to resources they did not request will be informed that this was done because resource was oversubscribed

• All changes from initial to final allocations will be documented and made available to the RP’s & GIG (i.e. both the before/after spreadsheets) .

12

Roaming Allocations

• TeraGrid Roaming allocations should be limited to the following three situations: –Startup and Education allocations, in which the user does not know

where best to run and may need to evaluate several architectures

–TRAC recommendations and awards made to encourage users to migrate to new or alternate platforms, especially in the case of overallocated resources

–Research allocation requests in which the proposal describes how the project will use roaming to conduct multi-site runs using unique grid capabilities and/or to minimize job waits (e.g. identifying least-busy resource).

• Failing this, the allocation should be made to the most appropriate resource(s), based on TRAC recommendations.

13

Improving documentation and communications

• Policies for proposal sections, length and content will be clarified, published and enforced. Proposals violating these guidelines will be immediately returned to proposers for revision and re-submission.

• Requirements for Startup proposals will be prominent in TeraGrid documentation, easily found by prospective first-time users.

• Requirements and evaluation criteria for novel resource classes (e.g. storage, advanced support) will be clarified, reviewed, and published for both requesters and reviewers.

• After every meeting, we will communicate via TG User News a summary of awards, and identify resources available for supplementary requests.

• Where resources are substantially underallocated, RP Directors will be empowered to make discretionary allocations until next quarterly cycle. – Substantial under-allocation is <80% of available resources

– In the 80-100% range, let allocated users get benefit of shorter queue waits

Next steps

• Want to ratify internal recommendations document and the TRAC Charter (available on wiki) in TG Forum meeting 7/22– Any discussion/edits should be sent to allocations-rat@teragrid.org by 7/16

• RAT will assign action items to appropriate people/WG’s to implement recommendations

• Sept meeting will be standard LRAC/MRAC but will implement most recommendations by that meeting and implement all changes by December meeting

14

RAT Members• SDSC: Richard Moore – chair, David Hart – the RAT MVP and the guy who

gets a lot of the action items for implementation• Indiana U: Craig Stewart – constructed User Survey and wrote its report,

Dennis Gannon, • LONI/LSU: Dan Katz• NCAR: Tom Bettge• NICS: Phil Andrews, Patricia Kovatch• NCSA: Tim Cockerill, Steve Quinn, John Towns• ORNL: John Cobb• PSC: Ralph Roskies – lead on section, Sergiu Sanielevici• Purdue: Carol Song, Bill Whitson• TACC: Jay Boisseau, Kent Milfeld – the guy who gets most of the action

items for implementation• UC/ANL: Joe Insley• Also thanks to Steve Meacham and Abani Patra for many comments and

suggestions

15