TERA Server Architecture

Copyright © 2010 Bluehole Studio, Inc.

How to Support an Action-Heavy MMORPG

from the Angle of Server Architecture

30 JUL 2010

Seungmo Koo (具升谋)

[email protected]

CGDC 2010

TERA’s Server Architecture 2

Table of Contents

Introduction

– TERA

Warming Up

Architecture

– TERA Server

Test

Summary


T.E.R.A.

The Exiled Realm of Arborea

– Open-beta test within this year

TERA via Mass Media

– Blockbuster MMORPG using Unreal Engine 3

– Development cost about 200,000,000 RMB during 3.5 years

– Various contents on the Seamless world

– Non-targeting, Non-targeting, Non-targeting, …

TERA via Developers’ View

– Is it possible?

• (X) Converting Action-heavy console game into MMORPG

• (O) Making MMORPG with strong action elements

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Action-Heavy MMORPG

Realistic Combat

– Dodge or block by observing the attacks of enemies

– Control the direction and distance to attack enemies

– Natural in Console or MO games

– High expectations in regard to high sync rate and low latency

Non-targeting Combat System

– A way to support action-heavy combat

Server-side Support for Action-Heavy Combat

– Space and collision calculation very rapidly

– Send and receive packets much more

Work load proportionate to the level of action-heaviness

• Introduction

• Warming Up

• Architecture

• Test

• Summary


T1

In Fact, 3D Volume

for Collision Check

Targeting vs. Non-Targeting

T6 T5

T4 T3

T2

Player

Ally

Enemy

Weapon

Direction

Non-targeting Targeting

Optimized space search and collision check are needed

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Terminology

Channel – The space where players actually explore

Continent – Literally, a large area of land having one or more channels

World – Server contains a group of continents

Planet – Group of worlds

– (aka) Realm server

– The unit of user clients’ view

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Server Architecture Requirement

Action-Heavy Combat

– Non-targeting battle in the overall combat system

• Increasing space-search and hit-judgment

Seamless World

– No loading and teleport within a world

– No limitation to deploy contents anywhere

• Creature, hunting-ground, and any type of content

• Implementation should not restrict contents-deployment

Inter-Server Contents

– Competitive contents among planets

– Community contents

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Supporting Action-Heavy Combat

Frequent Space Calculation

– Object-search & Hit-judgment

• CPU-intensive work

– No physics engine in the server

• Simplify collision volume

– Using lock-free manner [4]

• Lock radically increase the CPU usage

Frequent Transmission

– Aggregate broadcast

• In inverse proportion to action-heaviness

– The more frequent send, the better action-heaviness

• Important to find an adequate value for aggregation

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Supporting Seamless World

Popular Thread Model for Zone-based World

– A dedicated thread to a particular content or area

• The decreasing complexity of source-codes

– Work load could converge on a specific thread

• Inappropriate for implementing a seamless world

To Make Seamless World

– TERA takes on (homogeneous) worker thread model [1]

– No dedicated thread that handles specific contents or areas

• One thread pool including symmetric worker-threads

• Somewhat difficult and complex to implement

– No blockings among threads via lock-free implementation [2, 3]

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Supporting Inter-Server Contents

Types

– Community contents

– Shared world contents

• Common instance-dungeon pool

• Battleground server

Simple Structure

– To reduce inter-server complexity

– To regard the shared world as clients’ own planet

– No handover of network connection

Scalability

– Easy to add a new world server with new contents

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Server Architecture History (1/2)

First Architecture

– Project S1 prototyping version

– Front Server: world container

• Support action-heavy combat on a seamless world

– Back Server: DB cache server

– Inappropriate to inter-server contents

Client Front Server

Back Server

Planet

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Server Architecture History (2/2)

Second Architecture

– Broadcaster

• To support community contents (chat-channels, party, …)

• To reduce world servers’ network I/O

– DB proxy

• To support data-sharing contents (auction, warehouse, … )

• Serialize and validate the DB tasks

– Complex: difficult and annoying to implement

Client

World Server

DB Proxy

Planet

Broadcaster

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Final TERA Server Architecture

Split by Role

– World server for CPU intensive tasks

– Arbiter server for I/O intensive tasks

• The role of “DB proxy + Broadcaster”

• In fact, DB proxy and Broadcaster have a lot in common

– Shared code, similar task-style

– Reduced the complexity of server architecture

Client World

Server Arbiter

Server

Planet

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Arbiter Server

Single System Image [6]

– The only server that game-clients access

• Originated in Starcraft unit

– No handover of network connection

– Delegated for packet-broadcast by World servers

• Dispersing network I/O load

– Handling common contents among World servers

• Chatting, guild, party, …

Implementation Features – Using locks for common contents

– Two thread pools

• Blocking I/O (e.g. DB operation)

• Non-blocking I/O (e.g. packet broadcast)

• Introduction

• Warming Up

• Architecture

• Test

• Summary


World Server

Seamless World Container

– Space where players explore

• A world includes multiple continents

• Specialized for space search and assessment of the combat

– Sharable among multiple Arbiter servers

• Battle-group, common instance dungeon pool

Implementation Features

– Single thread pool consists of symmetric worker-threads

• No thread that exclusively handles particular contents

– Robust code base for multi-threading [2, 3]

• Making all tasks non-block-able through lock-free manner [4]

• (c.f.) Proactor [1] & Active-object pattern [5]

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Scalability

Server Topology – Can attach World servers to a single Arbiter server

– Convenient to arrange continents

• Different combinations of continents in multiple World servers

World Server Side – Identical worker-threads as the number of CPU cores

• Direct benefits from CPU core count and speed

Arbiter Server Side – Possibility of bottleneck

• Less critical of an issue than allocated bandwidth for service

– Utilizing the surplus CPU power

• Packet optimization to reduce bandwidth

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Additional Issues

Considering Contents Deployment

– Inter-server contents influence how to deploy World servers

– Considering the stability

• Can keep a Planet live even if a specific World server down

• Quarantine unstable contents (e.g. latest patched)

Split Servers Depending on Purpose

– Dedicated server for users’ convenience

• Chatting, searching, and so on

Order of Priority

– Not to put restriction on elements of fun

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Test with Dummy-Clients

Sisyphus – Client program operating dummy-bots for server-load test

• Operating 1000~1500 bots per one Sisyphus

• Recording real-players’ behavior patterns

WAN Simulator – Give virtual latency to mimic real-network environment

• Randomized latency around 200ms

• Required high performance machine

• Required dedicated line for load-test

Sisyphus Planet WAN Simulator Sisyphus

Sisyphus Sisyphus

Dedicated-line for test

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Stress Test

3rd Closed Beta Test

– Last Spring in Korea involving about 24000 users

– Events designed to complement daily stress test

• A sizable number of concurrent users were provided

• Server-status goes on smoothly throughout the stress test

Server Status

– Very low CPU usage on World servers

– Not any congestion handling network I/O on the Arbiter server

– Monitoring via VTune [7]

• Latest version to support Nehalem CPU [8]

• Introduction

• Warming Up

• Architecture

• Test

• Summary


Summary

Impossible is Nothing

– Not easy to make non-targeting battle in a seamless world

– But, it is certainly possible

Server Architecture Case by Case

– Optimized for each different type of MMOG

– Arbiter-World architecture for TERA

Complexity

– Support the “simplicity first” policy

– For stability, ease of management, and …

• Introduction

• Warming Up

• Architecture

• Test

• Summary


References

[1] Pattern-Oriented Software Architecture: Patterns for Concurrent and Networked Object Douglas Schmidt et al. John Wiley & Sons, 2000.

[2] Thread-Specific Storage for C/C++ Douglas Schmidt et al. C++ Report, 1997.

[3] The Art of Multiprocessor Programming Maurice Herlihy et al. Morgan Kaufmann, 2008.

[4] Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms Maged M. Michal et al. ACM symposium on Principles of distributed computing, 1996.

[5] Active Object: An object behavioral pattern for concurrent programming R. Greg Lavender et al. Pattern languages of program design, 1996.

[6] Single System Image R. Buyya et al. The international journal of high performance computing applications, 2001.

[7] VTune: Performance Analyzer Intel Software Network. http://software.intel.com/en-us/intel-vtune/

[8] Nehalem micro-architecture http://en.wikipedia.org/wiki/Nehalem_(microarchitecture)


감사합니다

谢谢!

Thanks!

Q & A

Technology

TERA Server Architecture