Upload
brian
View
63
Download
0
Embed Size (px)
DESCRIPTION
NetGrove: New Tools for Network Optimization. Tomas Vykruta Software Development Engineer XNA Publishing Platform Microsoft. Speed of Light Still Constant. Latency. Speed of light = 186,282 miles per second Seattle to L.A. 960 miles = 5 milliseconds Seattle to England - PowerPoint PPT Presentation
Citation preview
NetGrove: New Tools for Network OptimizationTomas VykrutaSoftware Development EngineerXNA Publishing PlatformMicrosoft
Speed of Light Still Constant
Latency• Speed of light = 186,282 miles per second• Seattle to L.A.
• 960 miles = 5 milliseconds• Seattle to England
• 4,799 miles = 26 milliseconds• Speed of light in fiber or copper slows to 60%• Each router adds 5 to 50 ms• DSL or cable modem adds 10 ms
Bandwidth on the Rise
Thanks, Bungie!
Also on the Rise• Player expectations
• More players in a session together• More voice, video, and background
downloads• Network-using features in your game,
such as data-feeds and spectator modes
• Network-using features in the platform
• Other home devices consuming bandwidth
Building a successful title• Optimizing game networking is still
very important!• Have a Plan:
1. Define max bandwidth goal2. Monitor bandwidth during development
**DO NOT LEAVE UNTIL SUBMISSION**
Defining Maximum Bandwidth• Host = worst case scenario• Let’s target top 80% of users
• 80% have 175ms or better latency• And 176kb/s down, 226kb/s up or better
bandwidth• From “The Real World: Console Latency
and Bandwidth on Xbox LIVE” white paper on xds.xbox.com
• Subtract reservation (i.e. voice chat)• Divide remaining bandwidth by “max # of
players – 1”
Bandwidth Calculation
HOST
CLIENT
CLIENT
CLIENT
CLIENT
56kbps
14kbps
14kbps
14kbps
14kbps
17kbps
17kbps
17kbps
17kbps
5 playergame
68kb
ps
Max Bandwidth cont’d• Scenario: theoretical 16 player game with
30 kbps reservation for voice chat• Host is updating 15 players
• And receiving data from 15 players• Max Bandwidth = 9 kbps down, 13 kbps
up per client max
• Also simulate 175 ms latency for playtests• Monitor regularly!
Max Bandwidth Worksheet Host Bandwidth 175 kbps Host Bandwidth 226 kbps Reservation 30 kbps Reservation 30 kbps Effective 145 kbps Effective 196 kbps
Players Maximum Kbps Players Maximum kbps2 145.00 2 196.004 48.33 4 65.338 20.71 8 28.00
12 13.18 12 17.8216 9.67 16 13.0720 7.63 20 10.3224 6.30 24 8.5228 5.37 28 7.2632 4.68 32 6.3264 2.30 64 3.11
Bandwidth Data 2007-2008Table 1: 2007 Latencies Table 2: 2007 BandwidthPercentage of Consoles with
this Latency or LessRound Trip Latency Between
Consoles (ms)
10% 3220% 4530% 5740% 7150% 84 (median)60% 10270% 13080% 17590% 25095% 32097% 38098% 43099% 54099.5% 64099.9% 770
Percentage of Consoles with this Bandwidth or Less
Downstream Bandwidth Between
Consoles (kb/s)
Upstream Bandwidth Between Consoles (kb/s)
0.1% 20 240.5% 40 441% 64 642% 72 803% 80 885% 104 12010% 144 17620% 176 22430% 200 25640% 248 28050% 336 (median) 352 (median)60% 480 52870% 904 92080% 1760 222490% 5184 5544
Latency and Bandwidth Data• 2008 bandwidth stats same as 2007• Median latency 84 ms• Only 10% of consoles are 32 ms or less• Worst case latency 500 ms• 336 kb/s median downstream bandwidth• 352 kb/s median upstream bandwidth• Typical packet loss 2%• Worst case packet loss 10%
The Real World• From real-world title network reviews
• Best case = 4 kbps per client in AAA FPS title
• Worst case = 370 kbps+ for the host in a 4 player game of a different AAA title!
From a real network review• “Stopped moving” message sent every frame• Many invisible objects simulated• Entire animation state sent (100 bytes+)• Uncompressed data• Position + Orientation + LookAt sent each frame =
12 + 36 + 36 bytes = 84 bytes• Host not coalescing data• NPCs simulated like players• Many more…• End result after optimizations = 5 kbps per client!
Run-time Bandwidth Monitoring• Fps, Memory Usage, why not peak
bandwidth?• Display on screen, always present• Visual alert to indicate packet loss
Tools• Instrumentation in game code• Subjective playtest experience• Packet sniffing
• Tried and true• Black box and low barrier to entry• Many sophisticated sniffers available
Network Monitor 3.2• Process Tracking to identify rogue applications• New engine for improved capture rate in high-
speed networks• Find conversations to view TCP streams, HTTP
flows etc. • Extensive parsers for over 300 protocols! • Network Monitor API: Create your own
applications that capture, parse, and analyze network traffic
Network Monitor 3.2
Quick NetMon Tour• Conversations• Find frame (CTRL+F)• Display and capture filters
• Declarative language with Intellisense• Color filters• Ex: XSP (by itself) filters down to Xbox LIVE
traffic• NMCAP for command-line captures
NetMon Protocols• Extensive script-based protocol
parser language• Documented in NetMon\Help\ParserLanguage.doc
• Create your own!• Trade with your friends! • Sure would be nice if the LIVE
protocol…
LIVE Secure Packet Parsing• Xbox 360 SDK and Games for Windows – LIVE SDK
both provide a NetMon Parser• XSP.NPL
• Automatically installs with XDK• Encrypted data is still encrypted
• XNET_STARTUP_DISABLE_PEER_ENCRYPTION to send unencrypted data while debugging
• XSP parser + NetMon filtering + no encryption =
see what’s actually happening on the wire
DISABLING ENCRYPTION• XNET_STARTUP_BYPASS_SECURITY added in
November 2008• Disables encryption, view unencrypted XSP
data in NetGrove and NetMon• Flag must be enabled on all consoles
• Silently disabled on RETAIL hardware• Stop using XNET_STARTUP_BYPASS_SECURITY
• Disables authentication and XSP completely
LIVE Server Traffic
Traffic to LIVE Statistics / Profile Server
Traffic to LIVE Presence Server
PIX – System MonitorPIX providescounters for LIVE servertraffic Traffic to LIVE Statistics
Server
So What Do You Do with This?• Debug pathological network conditions
• Excessive or unexpected LIVE traffic (TCR)• Mapping LIVE API calls to actual network traffic
• Optimize and tune?• Not really• No great analysis features• Easy to lose serious problems in the weeds
Enter the NetGrove• Ships in the XDK as of March 2008
• Coming soon for Games for Windows – LIVE
• Uses capture files from NetMon 2.0+• Sophisticated analysis of capture files• Flexible and functional GUI• Warning system
NetGrove Key Features• Interactive network topology view• Packet traffic view• Enhanced packet information display• Graphical analysis of packet statistics• Text-based output and command-line
usage
NetGrove Window
Network Topology View
Packet Traffic View
Packet Information
Graphical Analysis of Packet Statistics
Warnings Window
Text-based output
But wait, there’s more!• Includes built-in tutorial and sample
capture file
So What Do You Do with This?• Seriously!• First• Some• Discussion• About• Network• Optimization• (An aside in many parts)
The Story of Goldilocks• The Goldilocks principle of packet size
• 50 < Payload Size < 1264• Coalesce!
• The Goldilocks principle of frequency• This frequency is JUST RIGHT: 15–20
packets/s• Varies slightly by game type
Other Naïve Optimizations• Avoid XSP padding!
• This includes using port 1000 • Avoid struct padding!
• VDP always, UDP rarely, TCP almost never
• Don’t send packets you don’t need to send• For example, don’t send muted voice
Slightly More Complex Choices• Do not lock send rate to framerate• Use peer-to-peer networking when
feasible• Don’t need authoritative server• Can’t rely on one server to take high CPU
and network bandwidth load• Definitely send voice peer-to-peer
Voice and Video• Always use peer-to-peer networking• Don’t send muted voice• Partition traffic by
• Proximity to player• Team channel• “Importance” rating
Compressing Data• Quantize data• Pack Booleans as bit flags• Use polar coordinates (2 parameters) rather
than 3D position (3 parameters)• Use random number seed to send
simulation state• Do not send strings!• For larger chunks, use lossless compression
Modern Implementations• Delta compression• Client-side prediction• Lag correction and interpolation• Contextual bandwidth scaling• Separate guaranteed and non-
guaranteed data
Sequential Packet Delivery• Use with great caution• Separate guaranteed and non-guaranteed
data into different packets and queues• Can cause serious lag even under ideal
network conditions when a packet is dropped
• Also applies to in-house reliable packet delivery protocols
HOST
Sequential Packet Delivery
CLIENT
PACKET 0PACKET 1PACKET 2PACKET 3PACKET 4PACKET 5
STALL
DROPPED
PACKET 1TIMEOUT
REQUESTRESEND PACKET 1
PACKET 1 (R)
GAME
Setup Diagram`
Development PCRunning NetMon
Hub
`
GFWL client
*Series of Tubes
*
Living With Latency• Embrace quantum uncertainty
• You can never know where an object is• Only where it used to be• ... and how fast it was moving
• The current state is a probability field• Is the cat alive or dead?
• Each player has their own parallel universe• Our goal is to keep the universes similar
• No red pill
Prediction Relativity• Remember the parallel universes• Send information relative to the
recipient• “Fired toward position (x, y, z)”
• What if players are in different places?• “Shot at Shawn, missed 10° to the
left”• Robust even if player positions differ
Prediction Paradox• Larger packets = less bandwidth• Position + velocity + controller input• More data makes prediction work
better• Allows lower packet send rate• Fewer packet headers
Setup• Download and install NetMon 3.2• Run NetMon as Administrator• Enable P-Mode• Use a hub! Switches and routers do
not work.
Profiling Your Title• Start with a 2 player capture, then
gradually add more players• Capture with voice and/or video
enabled• Capture lobby, eliminate redundant
traffic• Capture gameplay when all players
are standing still, look for redundant packets
• Capture worst-case scenario (heavy battle)
Automated Monitoring• If you already have an automated
nightly runtime “monkey” test• NetGrove.exe -report Results.txt
mygame.cap• Generates detailed report (text
format):
Outbound Inbound Voice Out Voice InSystem name pkt/s kbit/s pkt/s kbit/s kbit/s kbit/s-------------------------------------- ----- ------ ----- ------ --------- --------192.168.1.3 31.3 32.7 36.9 66.6 0.0 0.0192.168.1.2 37.0 66.7 31.3 32.7 0.0 0.0
NetGrove in Action
Real World Fail Case• Title is ready for final cert• Was only tested with ideal network
conditions throughout development• Tested with 100 ms+ latency for first
time• Terrible lag causing random deaths,
bad user experience• Too late to fix = negative reviews
Common Pitfalls• Common pitfalls from real title reviews
• Idle network chatter• Uncompressed data• Unused data• Complex state• High update frequency
(10–15 Hz is sufficient)• Network simulating invisible objects• Lack of prediction• Bugs
Resources• Network Performance Optimization
white paper• Networking Best Practices white paper• Network Monitor 3.2 Connect Site
© 2009 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
http://www.xna.com