Upload
matthewturk
View
133
Download
1
Embed Size (px)
DESCRIPTION
The yt project ( http://yt-project.org/ ) is a community developed analysis and visualization system for astrophysical simulation data. In this presentation I talk a bit about what yt is, and then discuss the challenges and strategies for growing a community of practice.
Citation preview
The yt Project:Growing & Engaging a Community of Practice
Matthew TurkColumbia University
What is yt?
astro-ph/1011.3514astro-ph/1112.4482
yt-project.org
yt is a platform for analysis and visualization of astrophysical* simulations
install script:
Full dependency stackSource code
Development environmentGUI
Sample data
There are many simulation codes.
There are many simulation codes, but there’s only one sky.
data structures,methods,
assumptions,IO formats,
units,variable names,
...
Fully-Supported Semi-Supported In-Progress
EnzoFLASHOrionNyx
Uniform Data
ChomboAthena
ARTRAMSES
GDFPiernik
CactusHydra
Gadget
yt is designed to address physical,not computational,
entities and questions.
Orthogonal RaysNon-orthogonal Rays 1D
SlicesOblique SlicesProjections
2D
SpheresRectangular PrismsDisks/CylindersInclined BoxesClumpsExtracted RegionsEllipsoids
3D
Scripting interface, HTML5 GUI, in-situ processing, transparent multi-level parallelism...
yt is supposed to get out of the way.
from yt.mods import *pf = load(’galaxy0030/galaxy0030’)p = SlicePlot(pf, 2, ’Density’, ’c’, (200,’kpc’))p.save(’Galaxy’)
10-12 g/cc
6 C.B. Hummels et al.
Figure 1. Sample projections from the medium feedback and cooling suppression run at z = 0.5. Each image represent a column-densitymap of a region with 500 comoving kpc on a side. The white circle denotes r200 = 160 kpc.
tributions of a wide variety of atomic species, observable inabsorption against background quasars. We initially restrictourselves to low redshift, as there exists a substantial bodyof observations which link absorption strength to projecteddistance from the galaxy. In the following sections, we firstexamine the column density distributions, then explore thephysical properties of the gas giving rise to this absorption,and finally investigate the redshift evolution of the predictedabsorption strength.
3.1 Radial Profiles for Atomic Species
In this study, we examine the radial column density distri-bution for a number of absorption-line-generating species,well-sampling the range of ionization energies (and thus tem-perature/density regimes) of the CGM as shown in Table 2.At the two ends of the spectrum, the H I traces the cold,dense clouds, whereas the O VII probes the hot coronal gas.Between these regimes is a continuum of layers of mate-rial acting as the interface between these two phases. Thislayering e!ect takes place on many di!erent spatial scalesfrom small star-forming clumps of cold gas up to the galacticscale. Figure 1 demonstrates this at the latter scale, wherethe cold H I of the inner halo is enshrouded in warmer CIV-bearing gas, which is further encased and extended bygas exhibiting O VI. This provides us with some intuition
for understanding the following galactic profiles for thesespecies.
We produce radial profiles of the median gas columndensity in nine di!erent ionic species as a function of pro-jected radius extending from 2 kpc to 1 Mpc for our simu-lated galaxies. We probe this gas from the disk-dominatedregion (r < rinner ! 20 kpc) through the CGM (rinner <r < r200) into the intergalactic medium (r > r200). Figures2, 3, and 4 present these results (Figure 5 shows total gascolumn densities for comparison). The left and right sidesof each figure show the radial profiles of the same quantitybut for di!erent simulations. The left sides display a com-parison of simulations with and without di!erent forms offeedback: V, MF and MFCS, whereas the right sides exhibita progression of thermal feedback intensities: LF, MF andHF (the MF simulation is repeated on both sides to easecomparison). Each profile gives the median value expectedin an observation of a galaxy at a given radius over the red-shift range z = 0.1 " 0.5 (selected to approximately matchthe redshift ranges of the observations). To give some idea ofthe variation around the median absorber strength, in eachpanel we overplot a color band representing the quartiles ofthe column density distribution (i.e. 25% to 75%) for onemodel: MFCS (left) and HF (right). The quartile distribu-tion associated with the HF model is most indicative of thespread of all the other models, whereas MFCS produces a
c! 2012 RAS, MNRAS 000, 1–19
Canned Analysis Tasks
Absorption SpectrumCoordinate Transformations
Halo FindingMass FunctionsMerger TreesHalo Profiling
Level SetsLight ConesLight RaysTime Series
Star AnalysisTwo-Point Analysis
“Community”?
“Users”
“Developers”
Traditional View
“Users”
“Developers”
Most Scientific Software
“Devusers”
Community of Practice
“Developers”
“Developers”
Inspection and verification
“Developers”
Inspection and verificationTracking modifications
“Developers”
Inspection and verificationTracking modificationsSharing information
“Developers”
Inspection and verificationTracking modificationsSharing informationAdding functionality
“Developers”
Inspection and verificationTracking modificationsSharing informationAdding functionality
All are necessary characteristics of the scientific process as a whole.
“Users”
“Users”
Uncritical acceptance of code...?
“Users”
Uncritical acceptance of code...?
“These are people we give the code to that don’t care how it works.”
Developing as a Team
Challenges
Reward Structure
de facto and de jure
de facto and de jure
•Utilization of developed tools
•Respect from community
•Project involvements
•Invitations to speak
de facto and de jure
•Funding
•Publications
•Citation count
•Influence
Traditional astrophysics does not favor tool builders.
Chores
Chores
Documentation,testing,
outreach,infrastructure.
Tasks not fully-aligned with reward structure goals present great
motivational challenges.
Co-opetition
Co-opetition
•Funding
•Publications
•Citation count
•Influence( )
How developer community engagement -- as seen in cohesion, excitement, energy -- is
affected by funded improvements is unclear.
Strategies
The Art of Community by Bacon
Producing Open Source Software by Fogel
Team Geek by Fitzpatrick & Collins-Sussman
Organizing Simulation Code Collectives by Sundberg
Scientific Software Production by Howison & Herbsleb
You must design the community you want.
Technical& Social
Technical& Social
SVN hg
Repository
Users
Users & Repos
Users & Repos
Users & Repos
Users & Repos
Users & Repos
Users & Repos
Forky development:
very low barrier to entry; everything comes in the box.
Testing:
answer as well as integration tests get run every 30 minutes.
Code review:
eyes on (nearly) every changesetmentoring new developers
The path upstream must be obvious.
•Happy application
•Itch-scratching
•Submission
•Code review & mentoring
•Participation
Non-core contributed code:
a place for sharing scripts,data and images
Communication
Communication
All business is conducted openly.
Communication
Immediate
2 Communication
Low-Latency
Communication
High-Latency
Technical& Social
HRT
HumilityRT
HumilityRespectT
HumilityRespectTrust
HumilityRespectTrust
(Fitzpatrick & Collins-Sussman)
The most important thing I try to remember is that I want to foster a community of peers, not of elites.
I’ve noticed something is acting strangely with ...
““
I’ve noticed something is acting strangely with ...
You’re probably doing it wrong.““
““
Can you tell us how you would expect it to act?
I’ve noticed something is acting strangely with ...
““
““
I think that there might be a bug in the way ...
““
It’s like that for very good reasons. Don’t touch it.
I think that there might be a bug in the way ...
““
““
It behaves the way it does because ...
I think that there might be a bug in the way ...
““
““
Thoughtfulness,inquiry,
confidence,letting go.
This culture must be seeded directly.
This culture must be seeded directly,because culture will self-propagate.
letting go
Pride rather than ownership
Projects passing between people
Too much control can smother.
Too much control can smother.will
Successes
Developed by working astrophysicists.
Tom AbelDavid CollinsBrian CrosbyAndrew CunninghamNathan GoldbaumCameron HummelsJi-hoon KimSteffen KlemerKacper KowalikMichael KuhlenEve LeeChris MaloneChris MoodyAndrew MyersJeff Oishi
Jean-Claude PassyThomas RobitailleAnna RosenAnthony ScopatzDevin SilviaSam SkillmanStephen SkoryBritton SmithGeoffrey SoCasey StarkElizabeth TaskerMatthew TurkRick WagnerJohn WiseJohn ZuHone
Tom AbelDavid CollinsBrian CrosbyAndrew CunninghamNathan GoldbaumCameron HummelsJi-hoon KimSteffen KlemerKacper KowalikMichael KuhlenEve LeeChris MaloneChris MoodyAndrew MyersJeff Oishi
Jean-Claude PassyThomas RobitailleAnna RosenAnthony ScopatzDevin SilviaSam SkillmanStephen SkoryBritton SmithGeoffrey SoCasey StarkElizabeth TaskerMatthew TurkRick WagnerJohn WiseJohn ZuHone
“Usage of a HPC Data Analysis and Visualization System,” Szczepanski et al 2012
Usage on NSF NICS Nautlius
(play movie)
“... it seems likely that significant software contributions to existing scientific software projects are not likely to be rewarded through the traditional reputation economy of science. Together these factors provide a reason to expect the over-production of independent scientific software packages, and the under-production of collaborative projects in which later academics build on the work of earlier ones.”
Howison & Herbsleb (2011)
Thank you.
Tom AbelDavid CollinsBrian CrosbyAndrew CunninghamNathan GoldbaumCameron HummelsJi-hoon KimSteffen KlemerKacper KowalikMichael KuhlenEve LeeChris MaloneChris MoodyAndrew MyersJeff Oishi
Jean-Claude PassyThomas RobitailleAnna RosenAnthony ScopatzDevin SilviaSam SkillmanStephen SkoryBritton SmithGeoffrey SoCasey StarkElizabeth TaskerMatthew TurkRick WagnerJohn WiseJohn ZuHone