View
221
Download
3
Category
Preview:
Citation preview
Guided Content Optimization for Gears of War 3
Michael NolandEpic Games
michael.noland@epicgames.com
Overview
• Our iterative process and some tips/tricks• Memory feedback loop• Performance feedback loop• Texture streaming improvements• Wrapup
Information flow
• Capture and process data• Generate action items• Fix issues and retrigger captures
• Keep iterating– Streamline ID / capturing process for QA– Change information captured as new areas of
interest are identified
Current process
• Daily memory/perf standup• ~ Weekly reports on all maps• Quick turnaround on maps in flux• Periodic analysis of other captures– Continuous play-thru runs– MP soak tests– Cooked data report mining– Always loaded content packages
Implementation tips/tricks
• Classify dark memory– Ideally traceable all the way back to a placed actor
in the editor or a defined subsystem• Play thru maps on target as early as possible– Use artificial memory savers if necessary
• reducepoolsize, nosound• Report adjusted numbers to keep your eyes on the goal
– Establishes an ongoing baseline• BugItGo – Quick way to express a game location
Other data sources
• Always loaded content pulled in by script– Periodically scrape; savings here help everything
• Automatic cooked data reports– Top 10 lists showing large content items– Per level audio and animation breakdowns– ‘Content on-disc’ tags fed back to editor• Quickly narrow down which version is referenced
– Fresh data available every day
Memory feedback loop
1. QA plays thru individual levels capturing data2. QA runs a tool to process the raw data into
CSVs3. QA generates and sends out a hotspot report
showing the overall trend and specific dips4. Content or code makes changes and requests
an updated report from QA
1. Capturing raw data
• memleakcheck – Logs out:– Loaded objects– Streamed in levels– Various other stats / metadata– Can be done fast or full
• domemleakchecking 30– Does a fast memleakcheck every 30 seconds
1. Capturing raw data (cont.)
– Reboot console– Turn on memleakchecking and on-screen level
streaming report– Load a level and play thru it– Start playing the next level until all of the test level
finishes streaming out
2. Parsing raw data
• MemLeakCheckDiffer– Extracts information from unstructured .memlk– Loads a grouping script that defines how to bucket
objects together and in what order to report them– Two click processing for QA– Generates .CSV reports
3. Hotspot Report
• Lists lowest memory dips, with context– Loaded streaming levels– Commonly important bucket sizes• Static meshes• Skeletal meshes• Animation data• Sound data
– BugItGo location
4. Action Items
• Sent to a fairly broad list:– Level LDs / artists, lead LD– Strike team (memory/perf programmers and QA)– Producers and game team
• If dips are not well explained by an increase in LD buckets, someone will investigate further– Generated CSVs list all loaded assets– If still not accounted for, it’s dark memory that
needs to be fixed
4. Content Fixes
• Loaded levels list is often enough– Cinematic levels left loaded too long– Levels not streamed in/out at the right time– Expensive assets in a geographically large submap
• Transitions between maps can be problematic• Can look at a list of loaded assets– Merge/replace unique assets with library items– Stop non-visible assets from being pulled in– Actual cuts of noticeable content are fairly rare
4. Code issues
• May end up being a code problem or leak– Track genuine memory leaks with Malloc Profiler– Grow-only caches and pools are more common• Characterizing the expected amount is important
– Round-trip thru an exhaustive level and back to the menu
• Can factor into desired memory bar and ignore• Reset / purge when possible but plan for worst case
4. Code improvements
• Assets that are larger than expected– Re-evaluate your assumptions– A new game means new usage patterns
• Some real world examples– Static mesh collision data– Blocking volume brushes– Improving animation compression– Non-uniform light environment samples
Balancing Act
• Always consider memory savings in concert with performance
• Trading off memory for speed (in either direction) only helps if you have room in the other
Lessons from the past
• Went back to Gears 2 and retroactively pulled some stats from it to guide us thru Gears 3
• Full memlk runs for all SP maps– Gives approximate budgets, per-map slack
• Used as approximations, not hard budgets– New code optimizations give us more bang / MB– New features and fixed costs take away some too
Performance Feedback Loop
• Gather appropriate profiles– Run once to identify problematic areas and then
perform the appropriate profile captures – stat unitmax shows the time broken down into:• Render thread• Game thread• GPU time
• Generate action items
Performance Feedback Loop
– Capturing data• Capture on same run if the problem is steady-state• Otherwise, reload a checkpoint or start the level over
– Files generated depend on the bottleneck• Render thread or game thread
– Native CPU trace– Stats capture
• Game thread– Gameplay Profiler
• GPU– Native GPU trace (PIX)
Game thread
• Associated profiles– Gameplay Profiler captures– Stats captures– Native CPU traces
• Attack strategy– Depends on the actual bottleneck
• Animation, Line checks, Pathfinding, etc…
– Turn off expensive / low impact features– Optimize game code– Cut or modify triggering content
Gameplay Profiler
• Allows seeing script and native game code in context; great for tracking down hitches
Render thread
• Associated profiles– Native CPU trace– Stats Capture
• Most commonly an issue in split screen• Attack strategies– Reduce draw calls• Turn off bad shadow casters• Merge sections
– Code fixes for unexpected cases
GPU time
• PIX– Works like magic with a solid marker setup– We mark each drawcall with:• Material name• Mesh / Skelmesh name• Actor name• Hierarchy of rendering
– Prepass, Shadows, Opaque, Transparent, etc…
– Enabling shader PDBs
Texture Streaming
• Eliminating texture streaming artifacts• Two main kinds:– Insufficient streaming time• Enough memory but not enough time to fill it• (level loads / transitions)
– Totally over budget• Content uses too many unique textures
Texture streaming stats
• Desired texture memory is updated constantly• Log out stats via memlk or ustats to look for
over budget cases• QA also bugs locations with noticeable
blurriness or mip popping
Isolating streaming issues
• BugItGo to the bad spot on console or editor• Texture stats view shows which textures are
loaded and how many mips– Can be connected to a remote console
• Streaming system has a command to dump info on a particular texture– Lists everything it knows about a texture• Which objects reference it• Desired mip level calculation
What’s next
• Continuing to combat dark memory• Automatic capture file management– DB upload– Associated metadata
• Automated reporting after content checkins• Empowering users to look at lower level data
Conclusion
• Gather the information as early as possible• Filter it down into actionable information• Tighten the feedback loop• Keep iterating
References
• Many of the tools mentioned are documented in more detail on UDNhttp://udn.epicgames.com/Three/WebHome.html
• Hotspot reporting processhttp://udn.epicgames.com/Three/HotSpotReportGeneration.html
• UE3 performance and memory hubhttp://udn.epicgames.com/Three/PerformanceAndMemory.html
Recommended