1 #DynatraceCOMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

Asad Ali, @AsadThoughts

Java-Based Microservices

2 #DynatraceCOMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

Topics

• Overview of Producer Consumer Pattern

• Usage of Producer Consumer Pattern @ Dynatrace AppMon

• Microservices & Producer Consumer Pattern

3 #Dynatrace

Producer Consumer 2

Consumer 1

Consumer N

Work Queue

Producer Consumer Pattern

4 #Dynatrace

Producer Consumer 2

Consumer 1

Consumer N

Collection

Work Queue

Producer Consumer Pattern

5 #DynatraceCOMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

Producer Consumer @ Dynatrace AppMon

6 #Dynatrace

Dynatrace AppMon ArchitectureJava/.NET

PurePathCollector

dynaTraceServer

Web Server/ PHP

C++, VB, ADK

MQ/ESB

7 #Dynatrace

Metrics sent by agents

Visits

8 #Dynatrace

Producer Consumer 2

Consumer 1

Consumer N

Visit Store Collection

Work QueueCorrelation Engine

Real Time AnalyzerProducer Consumer Within AppMon

9 #Dynatrace

Producer Consumer 2

Consumer 1

Consumer N

Visit Store Collection

Work Queue

Work Queue Getting Full

Problem Symptom

10 #Dynatrace

Individual Transaction View

11 #Dynatrace

Individual Transaction View

12 #Dynatrace

46 threads are in Wait state

Thread Dump Analysis

13 #Dynatrace

Performance Report

14 #Dynatrace

Code Snippet

public synchronized void add(Visit visit) {Objects.requireNonNull(visit);super.add(visit);

}

public synchronized Visit get(int visitId) {return super.get(visitId);

}

15 #Dynatrace

Code Snippet

public synchronized void add(Visit visit) {Objects.requireNonNull(visit);super.add(visit);

}

public synchronized Visit get(int visitId) {return super.get(visitId);

}

Exclusive lock on visitstore caused consumers

to wait on each other

16 #Dynatrace

Optimized Code

public Visit get(int visitId) {final ReadLock readLock = rrwl.readLock();readLock.lock();try {

return super.get(visitId);} finally {

readLock.unlock();}

}

17 #Dynatrace

Optimized Code

public void add(Visit visit) {Objects.requireNonNull(visit);final WriteLock writeLock = rrwl.writeLock();writeLock.lock();try {

size.incrementAndGet();super.add(visit);

} finally {writeLock.unlock();

}}

18 #Dynatrace

Performance Report - Optimized

19 #Dynatrace

Performance Report - Optimized

20 #Dynatrace

Individual Transaction View - Optimized

21 #Dynatrace

Individual Transaction View - Optimized

22 #Dynatrace

23 #Dynatrace

Lesson #1

Analyze response time of your transactions

24 #Dynatrace

Lesson #2

public Visit get(int visitId) {final ReadLock readLock = rrwl.readLock();readLock.lock();try {

return super.get(visitId);} finally {

readLock.unlock();}

}

Parallelize reads

25 #Dynatrace

Lesson #3

Compare before and afterresponse time and throughput

26 #Dynatrace

Microservices & Producer Consumer Pattern

27 #Dynatrace

Producer Consumer 2

Consumer 1

Consumer N

Work Queue

Producer Consumer Pattern

28 #Dynatrace

JVM1

JVM

Producer

JMS

Producer Consumer With MicroservicesConsumer 1

JVM2

Consumer 2

JVMn

Consumer N

29 #Dynatrace

Producer Consumer With Microservices

JMS JVM1

Consumer 1

JVM

ProducerJVM2

Consumer 2

JVMn

Consumer N

30 #Dynatrace

Monolithic Application

31 #Dynatrace

2.68s Load Time

Monolithic Application

32 #Dynatrace

2.68s Load Time

94.09% CPU Bound

Monolithic Application

33 #Dynatrace

Can‘t scale vertically endlessly!

2.68s Load Time

94.09% CPU Bound

Monolithic Application

34 #Dynatrace

Frontendto Cloud

Scale Backendin Containers!

Service Approach

35 #Dynatrace

Go Live - 7am

36 #Dynatrace

Go Live - 12pm

37 #Dynatrace

What Went Wrong?

38 #Dynatrace

26.7s Load Time5kB Payload

Single Search Query End-to-End

39 #Dynatrace

26.7s Load Time5kB Payload

33! Service Calls

99kB - 3kB for each call!

Single Search Query End-to-End

40 #Dynatrace

26.7s Load Time5kB Payload

33! Service Calls

99kB - 3kB for each call!

Architecture ViolationDirect access to DB from frontend service

Single Search Query End-to-End

41 #Dynatrace

26.7s Load Time5kB Payload

33! Service Calls

99kB - 3kB for each call!

171! Total SQL Count

Architecture ViolationDirect access to DB from frontend service

Single Search Query End-to-End

42 #Dynatrace

The Fixed End-to-End Use Case

“Re-Architect” vs “Migrate” to Service-Orientation

2.5s (vs 26.7) 5kB Payload

43 #Dynatrace

The Fixed End-to-End Use Case

“Re-Architect” vs “Migrate” to Service-Orientation

2.5s (vs 26.7) 5kB Payload

1! (vs 33!) Service Call

5kB (vs 99) Payload!

44 #Dynatrace

The Fixed End-to-End Use Case

“Re-Architect” vs “Migrate” to Service-Orientation

2.5s (vs 26.7) 5kB Payload

1! (vs 33!) Service Call

5kB (vs 99) Payload!

3! (vs 177) Total SQL Count

45 #Dynatrace

46 #Dynatrace

Lesson #1

JMS JVM1

Consumer 1

JVM

ProducerJVM2

Consumer 2

JVMn

Consumer N

Pay attention to dependencies when using Microservices

47 #Dynatrace

Lesson #2

Measure inter-tier communicationtime for every transaction

48 #Dynatrace

Asad AliProduct Specialist

@AsadThoughts

http://blog.dynatrace.com

http://bit.ly/dtpersonal

49 #Dynatrace

Over To You

• Power of producer consumer pattern• Parallelize your reads• Measure before and after • Affect of Inter-tier communication in

Microservices• Do not ignore dependencies