1

1

Improving Perforce Performance

At Research In Motion (RIM)

Perforce User Conference April 2008

Tim Barrett

2

Agenda�RIM’s Environment�RIM’s Perforce performance problems�Project approach and major initiatives�

Memory upgrade�Server version upgrade�Move to Opteron/Linux�Read-only replica�RamSan solid state disk�

Minor initiatives�New initiatives since end of Performance Project

2

3

RIM�Market leader in mobile communications�More than just the maker of the BlackBerry

Environment at start of Performance Project�1,950 active Perforce users (now 3,000 users)�Two depots�

Main depot has 80% of the volume �Server version 2005.2, many GUI versions�SunFire V890 with 8 dual SPARC IV (1.35Ghz)

4

RIM’s Perforce Performance Problems�Perforce performance was a growing concern in 2006�No tools to accurately measure performance�

Best tool measured # of concurrent processes every 5 minutes

1422251602 week avg

112275360Apr 23-27

172174960Apr 17-20

MinimumMaximumMedianAverage

3

5

RIM’s Perforce Performance Problems

Number of Processes Executing

0

50

100

150

200

250

0:0

3

0:5

8

1:5

3

2:4

8

3:4

3

4:3

8

5:3

3

6:2

8

7:2

3

8:1

8

9:1

4

10:0

8

11:0

3

11:5

8

12:5

3

13:4

8

14:4

3

15:3

8

16:3

3

17:2

8

18:2

3

19:1

8

20:1

3

21:0

8

22:0

3

22:5

8

23:5

3

Time (HH:MM)

# o

f p

rocesses

Apr 26

6

Project Approach / Mandate�Performance Project Created�

Miscommunication led to the initial mandate: �Split the main depot into smaller depots to improve performance�

Mandate was later improved to remove the solution

from the mandate:�Improve performance for the Developers�

Project team later quantified the performance

improvement objective: �Deliver a 75% improvement in performance, adjusted for volume increases, based from the Developers perspective

4

7

Project Approach�The project approach was simply:�

Measure actual performance and determine what is

causing the performance problems at RIM�Utilize experience and advice from Perforce and peer

companies�Small improvements early in the project to show movement and commitment�Open, honest, frequent communication to all Perforce

users

8

Performance Monitor Script�RIM built a performance monitor script�

Focus on performance from Developer experience�5 processes executed every 15 minutes�Lapse time from first command to last command tracked

0.8s125.3s4.1s12.2sAverage

0.7s123.2s5.9s14.4sApr 23-27

0.7s209.6s5.0s18.1sApr 17-20

0.9s98.1s3.2s8.7sApr 9-13

0.9s49.0s1.4s4.7sApr 3-5

MinimumMaximumMedianAverage

5

9

Performance Monitor Script

00:00.00

00:17.28

00:34.56

00:51.84

01:09.12

01:26.40

01:43.68

02:00.96

02:18.24

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

Apr 26

Apr 27

10

Perforce Log Parser�Built advanced parser of Perforce logs to:�

Determine actual performance, volume, GUI versions�Identify what was causing performance problems at

RIM – both long term and targeted�Information dumped to a database and used in

reporting and ad-hoc analysis

6

11

Perforce Performance Issues�Performance problems at RIM were not unique�Perforce performance is impacted by cross-table locking�

Reads hold shared locks�Writes hold exclusive locks�

No one command or type of command generally causes the performance problems�

The combination on LONG READS followed by WRITE commands causes the log jam

12

Table Locking Problem - 1 table

TIME 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

LEGEND

Active Reader

Active Writer

Blocked Reader

Blocked Writer

7

13

Solutions to address performance�There are two ways to improve performance�

Improve speed�Increase the speed of every command to reduce the time locks areheld�

Reduce volume�Reduce the occurrence on inter-table locks�

Advice from Perforce and other peer companies:�Better hardware�Newer Perforce software (server & GUIs)�Improved configuration of Perforce

14

Major Initiative: RAM Upgrade�Starting configuration:�

32GB RAM for 1,950 users�Recommendation�

RAM should be 32MB per user or RAM should be 1.5KB per file�File cache affects speed as much as the amount of RAM�

RIM tested�32GB vs 64GB, 12% file cache vs 50% file cache�

Test results showed: �11% performance improvement with 64GB and 50% file cache over current 32GB with 12% file cache

8

15

Major Initiative: RAM Upgrade

Min Lapse Times of 5 Test Processes

2:30am - 7:30am EST

00:00.00

00:00.09

00:00.17

00:00.26

00:00.35

00:00.43

00:00.52

00:00.60

00:00.69

00:00.78

00:00.86

Apr

17

Apr

18

Apr

19

Apr

20

Apr

21

Apr

23

Apr

24

Apr

25

Apr

26

Apr

27

Apr

28

Apr

30

May

1

May

2

May

3

May

4

May

5

May

7

May

8

May

9

May

10

Date

Lap

se T

imes (

H:M

M:S

S)

Memory Upgrade

16

Major Initiative: RAM Upgrade�Results:�

Implemented April 29, 2007�Coincided with addition of 300 new Developers�Improvement NEVER noticed by Developers�

Speed improvements as expected in middle of the night�Speed improvements lost in daytime volume

9

17

Major Initiative: Upgrade server�Recommendation from Perforce�

Upgrade server from 2005.2 to 2006.2�Known performance problems with 2005.2 �

RIM tested various commands against both

versions in the same environment�Most commands had improved performance�Three commands were consistently slower�

No explanation was found by RIM or Perforce

18

Major Initiative: Upgrade server

-14.1%44s38sOPENED

-5.7%267s252sFSTAT

-4.9%185s176sCHANGES

21.6%666s850sSYNC

69.4%2246s7335sSUBMIT

10.5%85s95sLABELS

5.2%4121s4347sINTEGED

2.4%161s165sFILES

80.5%64s329sDIRS

Variance2006.22005.2Average times

10

19

Major Initiative: Upgrade server

00:00.0

00:17.3

00:34.6

00:51.8

01:09.1

01:26.4

01:43.7

02:01.0

02:18.2

9:00

9:30

10:00

10:30

11:00

11:30

12:00

12:3

0

13:00

13:30

14:0

0

14:30

15:00

15:3

0

16:00

16:30

17:00

17:30

18:00

May 14

May 15

May 16

May 17

May 18

00:00.00

00:17.28

00:34.56

00:51.84

01:09.12

01:26.40

01:43.68

02:00.96

02:18.24

9:00

9:30

10:00

10:30

11:00

11:30

12:00

12:30

13:00

13:30

14:00

14:3

0

15:00

15:3

0

16:00

16:3

0

17:00

17:3

0

18:00

May 28

May 29

May 30

May 31

June 1

Before the upgradeBefore the upgrade

After the upgradeAfter the upgrade

20

Major Initiative: Upgrade server�Results:�

Server upgrade completed May 19, 2007�Immediate improvement noticed by Developers�

“Still slow but useable”�Improvements allowed RIM Developers to work through

the summer�Not the final solution!

11

21

Major Initiative: Opteron/Linux�Recommendation from Perforce and peer companies�

Move from Sparc/Solaris to Opteron/Linux �RIM tested the following platforms:�

Current: SunFire V890 with 8 dual SPARC IV (1.35Ghz) CPU, 64GB RAM, Solaris 9�Proposed: SunFire x4600, 8xAMD Opteron model 8220 processor (2.86Ghz dual-core), 128 GB RAM, Linux (REHL4, update5)�

Test results showed improvement from 62% to 98%

22

Major Initiative: Opteron/Linux

79%Average

84%1h20m8h31mCheckpoint

62%1h45m4h4mRebuild from Checkpoint

73%17s1m3sSubmit Integrate Test

98%31s29m54sIntegrate Test

72%38m8s2h14m51sPopulate Test

85%39s4m6sLock Test

VarianceOpteron/LinuxSparc/Solaris

12

23

Major Initiative: Opteron/Linux

Average lapse time of 5 test processes

M-F 09:00-18:00

00:00.0

00:04.3

00:08.6

00:13.0

00:17.3

00:21.6

00:25.9

00:30.2

00:34.6

JUL 30-

3

AUG 7-

10

AUG

13-17

AUG

20-24

AUG

27-31

SEP 4-

7

SEP

10-14

SEP

17-21

SEP

24-28

OCT 1-

5

OCT 9-

12

OCT

15-19

OCT

22-26

Lap

se T

ime (

MM

:SS

.0)

Average Lapse Time of 5 sample processess

24

Major Initiative: Opteron/Linux�Results:�

Implemented September 22, 2007�Results immediate and dramatic�

All Developers noticed improvement�Could have been the final solution……but wasn’t!

13

25

Major Initiative: Opteron/Linux

00 : 01 .73

00 : 02 .16

00 : 02 .59

00 : 03 .02

00 : 03 .46

00 : 03 .89

00 : 04 .32

AV

ER

AG

E

26

Major Initiative: Read-only Replica�Recommendation from peer company�

Move “read only” build commands to mirror replica�RIM modified P4 JREP script from Perforce based on

suggestions from peer companies�RIM is in the process of moving all “build” activity from

main server to the replica�Build activity accounts for 60% of the total volume on Perforce every day�Top 15 build accounts responsible for 98% of the p4 sync volume

14

27

Major Initiative: Read-only Replica

Results:�Implemented MIRROR on new server Fall 2007�

Some Development builds moved to mirror�Better performance than main server�But, problems developed with consistency�

Implemented MIRROR on the main server March 2008�Eliminated inconsistency�Now working at moving over all build activity

28

Major Initiative: RamSan�Recommendation from peer company�

RamSan dramatically improves the speed of Perforce�RIM tested 128GB RamSan solid state disk against:�

Local Disk – SCSI Raid 0�Production Array – SUNSAN Array 9985�

Test results were mixed�Low Level testing showed improvement with RamSan over Sun Array�Real world testing showed no noticeable improvement

15

29

Minor Initiatives�“Kill” orphaned processes�

Many orphaned processes cluttered the server�No performance improvement proven�

Frequent Rebuild from Checkpoints�Minor relief on version 2005.2�No performance improvements in 2006.2�Used every 6 months to reduce Checkpoint time�

Optimize Protection table�Based on advice from 2007 User Conference�Expected improvements impossible to measure

30

Minor Initiatives�Upgrade GUI versions�

Many Users running old versions of GUI�Some performance problems are related to Clint GUI�

Set Minimum and Recommended levels�Min – 2005.2�Recommended – 2006.2�

Requested clients to upgrade to current versions�Mixed results�

Eliminated P4 EXP from install package

16

31

Client GUI Versions: March 2008

32

Minor Initiatives�Reducing client volume�

Requested polling to be turned off or reduced�Local Proxy Investigation�

Local proxy proven to worsen performance�Limit scope of Clientspecs�

Reduced accidental “syncing” of the entire depot

17

33

Minor Initiatives�Divide the depot�

Time spent investigating and build process to divide the

depot along functional lines�Abandoned due to problems with the process,

complexity of our environment, inability for other tools

to support multiple depots�Archive old version files�

On hold waiting for Perforce to present a solution

34

Timeline

FEB MAR APR MAY JUN JUL AUG FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DECSEP OCT NOV DEC

Depot Depot CleanClean--upup

Depot Depot CleanClean--upup

Depot Depot CleanClean--upup

Depot Depot CleanClean--upup

Depot Depot CleanClean--upup

Rebuilt Rebuilt IndexesIndexes

Orphan Orphan Process Mgmt Process Mgmt

ToolTool

Install Install MirrorMirror

Upgrade Upgrade MemoryMemory

Upgrade Upgrade Server/GUI Server/GUI SoftwareSoftware

Convert to Convert to Opteron/ LinuxOpteron/ Linux

Investigate Investigate RAMRAM--SANSANPerformance Performance

Measurement Measurement ToolsTools

Optimize Optimize Protections Protections

TableTable

Notify Notify “problem” “problem”

usersusers

18

35

The Future of Perforce at RIM�More improvements are always required�

Volume will erode current improvements�Need to optimize remote sites�Focused on improving processes with Perforce�

Initiatives:�Investigate better build, branching, development methodologies�Aggressively improve remote performance�Implement P4 Broker�Better monitoring tools�Perforce Archive?�Upgrades!

36

The Next Server Upgrade�RIM is always looking to upgrade our applications

and tools�Tested our current version of 2006.2 vs 2007.2 and 2007.3�Results showed little performance improvement in 2007.2�Results showed some expected improvements in 2007.3�Planned upgrade to 2007.3 (Server and Client) May 3, 2008

19

37

New Tests (Feb 2008): Upgrade server

31.4%108s118s157sSUBMIT

18s

24s

38s

67s

59s

388s

33s

13s

2006.2

-111.1%37s28sOPENED

8.96%22s24sFSTAT

-0.2%38s36sCHANGES

-1.2%68s75sSYNC to CL

5.4%56s68sSYNC

-0.5%390s375sINTEG

14.2%28s29sFILES

-12.4%15s13sDIRS

Variance2007.32007.2Average times

38

Summary�Performance project delivered a 90% improvement in

performance�Absorbed significant increase in volume at same time�

Comments from Developers:�“Wow, the depot is fast now.”�“I think you’re going to have a lot of happy developers – this one included”�“I’ve never seen Perforce perform this well! Great job guys!”�“I was just using Perforce and…it’s fast. Really fast. I love it.”

20

39

Questions?