Upload
others
View
14
Download
0
Embed Size (px)
Citation preview
Folie 1Vortrag > Autor > Dokumentname > Datum
Interactive Visualization of Large Simulation Datasets
Andreas Gerndt, Rolf Hempel, Robin WolffDLR Simulation and Software Technology
EuroMPI 2010 Conference, Stuttgart, Sept. 13-15, 2010
Folie 2Vortrag > Autor > Dokumentname > Datum
Introduction
Folie 3Vortrag > Autor > Dokumentname > Datum
DLR German Aerospace Center
Research onAeronauticsSpaceTransportEnergy
Space AgencyProject Management Agency
Folie 4Vortrag > Autor > Dokumentname > Datum
Locations and employees
6500 employees across 29 research institutes and facilities at
13 sites.
Offices in Brussels, Paris and Washington.
Koeln
Oberpfaffenhofen
Braunschweig
Goettingen
Berlin
Bonn
Neustrelitz
Weilheim
Bremen Trauen
Dortmund
Lampoldshausen
Hamburg
Stuttgart
Folie 5Vortrag > Autor > Dokumentname > Datum
Earth observationNational data repositoryTechnology development
Left: SAR image of Cotopaxi volcanoAtmospheric and climate research
Visualization at DLR Applications in Experiment and Simulation
Planetary scienceSeveral space missions, e.g. Mars ExpressHigh-resolution Stereo CameraMulti-spectral data10 – 100 meters / pixel
Folie 6Vortrag > Autor > Dokumentname > Datum
Simulators for cars, aircraft, helicopters, …Left: Dynamic car simulatorResearch into driver assistance systemsVisualization of external view
Visualization at DLR Applications in Experiment and Simulation
Robotics / MechatronicsSpacecraft designOn-orbit satellite servicingAstronaut training
Folie 7Vortrag > Autor > Dokumentname > Datum
AerodynamicsCFD and experimentTurbines, airplanes, helicopters, spacecraftDevelopment of CFD codes (TAU, TRACE)Experiments used to validate simulations
Prominent DLR institutes specialized in CFDInst. for aerodynamics and flow technologyInstitute for propulsion technology
Visualization at DLR Applications in Experiment and Simulation
Folie 8Vortrag > Autor > Dokumentname > Datum
Research on software / computer science topicsCo-operations with engineering institutes in advanced software projectsMain topics include HPC and VisualizationVisualization projects with DLR institutes on
CFD around airplanes / in turbomachinesplanetary surface dataon-orbit servicing of satellites
Visualization at DLR DLR Simulation and Software Technology
Folie 9Vortrag > Autor > Dokumentname > Datum
Objectives of Virtual Reality (VR) Research
Important aspectsImmersionInteractivity3D VisualizationMulti-modality
Interactive CFD data exploration in a virtual environment
Folie 10Vortrag > Autor > Dokumentname > Datum
Overview
Problem Description
Pipeline Distribution / Parallelization Framework
Large-scale Dataset I/O
Multi-Resolution and Data Streaming
Co-Execution of Simulation and Post-Processing
Folie 11Vortrag > Autor > Dokumentname > Datum
Problem Description
Folie 12Vortrag > Autor > Dokumentname > Datum
Problem Description What is a „large“ simulation dataset?
Simulation of the air flowaround a complete transportairplane configuration
CFD calculations at the Institute for aerodynamics and flow technology with the TAU code
Solution of the Raynolds-averaged Navier-Stokes equationsHPC system: C2A2S2E (see next slide)
300-400 / 2000 mio. grid points (without / with acoustics)Test case: Slat track noise (LES)
80 mio. pnts., 6.7 GB, 10,000 time steps19 minutes / time step on 2048 cores
Folie 13Vortrag > Autor > Dokumentname > Datum
Problem Description The C2A2S2E Supercomputer
C2A2S2E Cluster (SUN) 16 compute racks, 48 blades each2 AMD Opteron quad core processors, 1.9 GHz (Barcelona)
768 nodes, 6144 cores46.6 TFlop/s Peak performanceTAU 1 core: 1 GFlop/s
all cores: 3 TFlop/s12288 GB Main memory
16 GB per node (758 nodes)32 GB per node ( 10 nodes)
SUN Mega SwitchParallel file system (GPFS)High speed link to Airbus Bremen (100 Mbit/s)Dedicated to aircraft research2010 performance update by factor 3
Folie 14Vortrag > Autor > Dokumentname > Datum
Problem Description First Experiences with Virtual Reality
2007: VR system set up at Institute for aerodynamics and flow technology
Powerwall 2.7m x 2.0m2 projectorsIR tracking system2 dual-core Opteron processorsVisualization software: Ensight Gold 8.2
ProblemsSystem cannot handle full-resolution datasetsVisualization is too slowMissing interaction metaphors
System mostly used for „Colorful pictures for visitors“
Folie 15Vortrag > Autor > Dokumentname > Datum
Problem Description CFD Simulation of a Turbo-Engine Compressor Section
Four-stage compressor sectionSimulation of the transition point between laminar and turbulent flow instationary problemComputational domain split into blocks (colored)123 Million mesh pointsNational Supercomputer HLRB-II (SGI Altix 4700) at the Leibnitz- Rechenzentrum in Munich
In total 15 Terabytes generatedCompute time: 210,000 CPUhExample is two years oldToday: Intention to run a problem three times as large
Folie 16Vortrag > Autor > Dokumentname > Datum
Pipeline Distribution
Folie 17Vortrag > Autor > Dokumentname > Datum
Pipeline Distribution Visualization Pipeline
Visual analysis of simulation data is organized as a post-processing pipelinePipeline input: large-scale grid-based CFD datasetFiltering stage processes the data
Produces more manageable data sizes (cutting, clipping, converting, merging, re-sampling, …)Extracts features of interest (isosurfaces, particle tracing, vortices, topology information, …)May contain multiple processing steps and can be organized as a data flow network (multiple inputs and outputs)
MappingCreates visualization objects
Rendering
Filtering
Dataset
ExtractedData
Mapping
VisualPrimitive
Rendering
ImageData
Folie 18Vortrag > Autor > Dokumentname > Datum
Pipeline Distribution Data Parallelization
Several parallelization approaches possibleBy nature of the post-processing pipeline, heavy work is usually located at the beginning of the pipeline
Data parallelization is the most efficient approach in most casesCommunication costs are reduced to a minimum
Folie 19Vortrag > Autor > Dokumentname > Datum
Pipeline Distribution Data Parallelization
Determine heavy work and parallelize these pipeline sectionsSplit data and allocate sections to parallel processesCombine partial results and execute remaining steps
Folie 20Vortrag > Autor > Dokumentname > Datum
Pipeline Distribution Distributed Post-Processing Framework
Distributed post-processing framework ViracochaUses data parallelization approach Has been developed in co-operation with the VR Group at Aachen University (RWTH)Message-passing best choice for parallel post-processingTCP/IP for communication in heterogeneous hardware systems
TCP/IP
Viracocha
Scheduler Worker
Message Passing
ViSTA FlowLib
ExtractionManager
Worker
VTK VTK
Folie 21Vortrag > Autor > Dokumentname > Datum
Pipeline Distribution Distributed Post-Processing Framework
Generic Viracocha layer architectureApplication-independentEasy integration of new functionsParallel concept (scheduler, workers) stays untouchedVisualization Toolkit (VTK) is used for most of the basic post- processing algorithmsAlgorithms from own research added to Toolkits
TCP/IP
Viracocha
Scheduler Worker
Message Passing
ViSTA FlowLib
ExtractionManager
Worker
VTK VTK
Folie 22Vortrag > Autor > Dokumentname > Datum
Pipeline Distribution Distributed Post-Processing Framework
ViSTA FlowLib is used as frontendOptimized for visualization of unsteady simulation dataInteractive visualization using Virtual Reality technologiesReal-time interaction by decoupling heavy data handling and post-processing work More features can be requested from backend without disrupting interactive exploration
TCP/IP
Viracocha
Scheduler Worker
Message Passing
ViSTA FlowLib
ExtractionManager
Worker
VTK VTK
Folie 23Vortrag > Autor > Dokumentname > Datum
Large-scale Dataset I/O
Folie 24Vortrag > Autor > Dokumentname > Datum
Large-scale Dataset I/O Viracocha Data Management System
Concurrent data management handles I/O loadCaching (primary / secondary)PrefetchingOptimized loading strategiesData services
Ask scheduler for next dataLoad balancing
TCP/IP
ViracochaScheduler Worker
Message Passing
ViSTA FlowLib
ExtractionManager Worker
Proxy ProxyServer
DataData Data
Folie 25Vortrag > Autor > Dokumentname > Datum
Large-scale Dataset I/O Domain Decomposition
Multi-block datasetsCan easily be distributed among several processesOne block per time level = one file on diskSufficient for cell-based (local) algorithms
Unstructured datasetsMake use of TAU’s partitionerOnline Monitoring: no I/O required
Blocks of a Propfan dataset,block-wise colored
Domain decomposition for a simulation using TAU
Folie 26Vortrag > Autor > Dokumentname > Datum
Large-scale Dataset I/O Loading on Demand
Global post-processing algorithmsE.g. particle integrationOnly block containing current particle position needs to be loaded into main memoryMetadata about block connectivity helps to identify next block when particle leaves current blockLoad on demand considerably reduces I/O load
Interactive particle seeding in an immersive environment
Folie 27Vortrag > Autor > Dokumentname > Datum
Multi-Resolution and Data Streaming
Folie 28Vortrag > Autor > Dokumentname > Datum
Multi-Resolution and Data Streaming Data Streaming
Decouple frontend and backendImprovement of interactivity within virtual environments
However: Extraction still needs timeSolution: Data streaming
Motivation: Progressive JPG >>>Immediate feedbackFast impression of the final result
Exploit first approximate resultsQuick start of data explorationPossible to abort running comp. and restart it with new parameters
ProblemAdditional communication and more computation timeAppropriate CFD algorithmsProgressive approaches hardly available
Sour
ce: h
ttp://
wp.
nets
cape
.com
/eng
/moz
illa/
2.0/
reln
otes
/dem
o/pj
pegd
emo.
htm
l
Folie 29Vortrag > Autor > Dokumentname > Datum
Multi-Resolution and Data Streaming Data Streaming
Multiple streaming steps of Lambda-2 vortex extraction computed by 4 workers and then streamed
independently to the visualization frontend
Folie 30Vortrag > Autor > Dokumentname > Datum
Multi-Resolution and Data Streaming Progressive Streaming
Analytical cut functionIntersections on cell edges have to be foundGeometry of cut mesh unknown in advanceFast
SamplingGeometry is already known everywhere in advanceCut surface can be visualized immediately Interactivity!Only scalars sampled at vertices are transmitted
Computed in parallel on backend
Triangulation parameters of a cut cylinder (right), resulting cut
through the propfan (top)
Folie 31Vortrag > Autor > Dokumentname > Datum
Multi-Resolution and Data Streaming Progressive Streaming
Progressive approachRealized as surface sub-divisionDyadic split: classical approach for triangle meshesSqrt-3 refinement
Fewer points to addSlower refinementSmoother streaming
Sqrt-3 refinement steps with special boundary treatment
3 3
Sqrt-3 refinement (Level 1)
Sqrt-3 refinement (Level 2),initial grid (red)
Folie 32Vortrag > Autor > Dokumentname > Datum
Multi-Resolution and Data Streaming Multi-Resolution Geometry
Adaptive renderingCache multi-level streaming data for Level-of-Detail switchIncrease frame rate by using different LODs selected by priorities
E.g. context geometry vs. extraction data
Handled by multi-resolution manager
Selective refinement Get viewpoint of the userClose parts of objects are rendered with higher resolutionRequires adaptive meshes
Main advantage of Sqrt-3 data structure
Approach can also be applied for polylines (e.g. particle trajectories)
T1
T5T2 T7
T3
T9
T10
T8
T4 T12
T6T13
T14T15
T11T16
T1
T2
T3
T4
T1 T2 T3 T4
T1 T2 T3 T4
T2 T7 T8T1 T5 T6 T3 T9 T10 T4 T11 T12
T6 T13 T14 T11 T15 T16
Sqrt-3 selective refinement
Folie 33Vortrag > Autor > Dokumentname > Datum
Multi-Resolution and Data Streaming Multi-Resolution Geometry
View-dependent optimized view of propfan blades, original with 340,000 triangles (left), view-dependent optimized with 8,532 triangles (mid), and
wireframe presentation of the entire view-dependent optimized model (right)
Folie 34Vortrag > Autor > Dokumentname > Datum
Co-Execution of Simulation and Post-Processing
Folie 35Vortrag > Autor > Dokumentname > Datum
Co-Execution of Simulation and Post-Processing Online Monitoring and Computational Steering
Challenges:Steer simulation and maintain data coherenceIntegration of interactive post- processing
Approach:Interactive post-processingIntegrate steering moduleAttach streaming visualization pipeline
Goal: Interact with the simulation during the run
Storage
Simulation
Visualization
Simulation
VisualizationSt
ream
ing
Visualization
traditional approach monitoring & steering approach
post
-pro
cess
ing
Ste
erin
g
post
-pro
cess
ing
Monitoring
Folie 36Vortrag > Autor > Dokumentname > Datum
Co-Execution of Simulation and Post-Processing Online Monitoring and Computational Steering
First results of prototypeMesh adaption while simulation runsProcessing done in parallel (shown by color)
original mesh refined mesh
Folie 37Vortrag > Autor > Dokumentname > Datum
Co-Execution of Simulation and Post-Processing Distributed Post-Processing Infrastructure
Goals: Exploit parallelism using various techniques and resourcesReduce amount of data communicated across pipelineRendering in parallel on GPGPU cluster (integrated in HPC environment)“Light-weight” frontend sufficient
Simulation
Bac
kend
Fron
tend
Storage
HPC-Resources
Render Cluster
Displays
Dataset
Filtering
Extracted Data
Mapping
Visual Primitive
Rendering
Image Data
Parallel Rendering
Task A Task D Task G
Task B Task E Task H
Task C Task F Task I
Task-Parallelism
Data-Parallelism
Folie 38Vortrag > Autor > Dokumentname > Datum
Co-Execution of Simulation and Post-Processing Distributed Post-Processing Infrastructure
InfinibandSwitch SC
Remote Locations
Login Node
GPU Compute/Render Nodes
Workstations SC-BS
Optional Infiniband Connection
Eth
erne
t Rou
ter
100TBGPFSAnton
DFN-Network
Intranet
EthernetSwitch
EthernetSwitch SfR
500TBGPFSCASE
InfinibandSwitch AS
200TBGPFS
AS
+50TB SC
Teamserver SC-BS
AntonCluster
C2A2S2ECluster
Infiniband > 10Gbit/s
Ethernet 100/1000Mbit
155Mbit DFN-Network
VR-Lab SC-BS
Planned Infrastructure at DLR:
Folie 39Vortrag > Autor > Dokumentname > Datum
Summary
Growing importance of visualization in large-scale numerical simulation
Today VR techniques mainly used for „pretty pictures“
Fast processing of large datasets required for interactive visualization HPC (parallel execution) for first stages in visualization pipeline
Close integration of application and postprocessing requires dedicated hardware / software infrastructure
Pure batch execution model for HPC no longer feasible
Interactive Visualization of Large Simulation DatasetsIntroductionDLR�German Aerospace CenterLocations and employeesVisualization at DLR �Applications in Experiment and SimulationVisualization at DLR �Applications in Experiment and SimulationVisualization at DLR �Applications in Experiment and SimulationVisualization at DLR �DLR Simulation and Software TechnologyObjectives of Virtual Reality (VR) ResearchOverviewProblem DescriptionProblem Description�What is a „large“ simulation dataset? Problem Description �The C2A2S2E SupercomputerProblem Description �First Experiences with Virtual RealityProblem Description �CFD Simulation of a Turbo-Engine Compressor SectionPipeline DistributionPipeline Distribution �Visualization PipelinePipeline Distribution �Data ParallelizationPipeline Distribution �Data ParallelizationPipeline Distribution�Distributed Post-Processing FrameworkPipeline Distribution�Distributed Post-Processing FrameworkPipeline Distribution�Distributed Post-Processing FrameworkLarge-scale Dataset I/OLarge-scale Dataset I/O�Viracocha Data Management SystemLarge-scale Dataset I/O�Domain DecompositionLarge-scale Dataset I/O�Loading on DemandMulti-Resolution and Data StreamingMulti-Resolution and Data Streaming �Data StreamingMulti-Resolution and Data Streaming �Data StreamingMulti-Resolution and Data Streaming �Progressive StreamingMulti-Resolution and Data Streaming �Progressive StreamingMulti-Resolution and Data Streaming �Multi-Resolution GeometryMulti-Resolution and Data Streaming �Multi-Resolution GeometryCo-Execution of Simulation and Post-ProcessingCo-Execution of Simulation and Post-Processing�Online Monitoring and Computational SteeringCo-Execution of Simulation and Post-Processing�Online Monitoring and Computational SteeringCo-Execution of Simulation and Post-Processing�Distributed Post-Processing InfrastructureCo-Execution of Simulation and Post-Processing�Distributed Post-Processing InfrastructureSummary