GlassFish Community Feedback

GlassFish usage in Rakuten Inc.

About me Name: Makito Hashiyama(@capyogu) Age: 29 Role: Team manager / In charge of APIs for Rakuten Ichiba Like: GlassFish, Tomcat, KVS(memcached, Coherence), GAE,

Android… E-mail: makito.hashiyama@mail.rakuten.com,

hashiyaman@gmail.com

What is Rakuten? E-commerce and Internet company based in Tokyo, Japan B2B2C e-commerce platform

E-Commerce

eBook

Travel

Other services & businesses

Rakuten Institute of Technology 　　

Development center

Head Office / Regional Headquarters 　

Head Office

How We Are Using GlassFish

Usage One of core API service for Rakuten Ichiba Require high availability

Environment Use on production environment GlassFish version 3.1.2.2

Our APIOracle

Coherence

GlassFish Clusters

External APIs

SOAP/REST

Client side

Reference implementation of Java EE Only GlassFish supported JAX-WS as standard in 2007 It was an advantage to evaluate new features earlier

Easy to manage with low cost We need to manage huge cluster without stopping Cost saving(Weblogic -> GlassFish)

Benefits gleaned from Glassfish

What Worked with GlassFish

Community support(create patches) https://java.net/jira/browse/GLASSFISH-5200

(If use jvmRoute JSESSIONID cookie is not secure even in HTTPS)

https://java.net/jira/browse/GRIZZLY-1333 (NetworkAddressValidator will fail when passed property substitution values)

We have contributed patched to GlassFish community.

Improvement to handle a huge traffic

Rakuten Super Sale Biggest online sales in Japan A lot of doorbuster deals(It causes a huge amount of traffic)

Performance bottleneck External APIs called by our API were slow down We needed to improve the system at the peek time

Our API

External APIs

delaySlow downClient side

RequestTask

Queue

Worker Thread

Worker Thread

Worker Thread

GlassFish

Worker Thread

Worker Thread

External APIs

delay

delay

delay

delay

delay

CPU load was high

Improvement to handle a huge traffic

Client side

RequestTask

Queue

Worker Thread

Worker Thread

Worker Thread

GlassFish

External APIs

(1)According to vmstat, ‘run queue’ was very high(2)Decrease worker threads to keep ‘run queue’ low(3)As a result, latency increased but throughput was improved

Improvement to handle a huge traffic

Client side

As a result… Our API could process over 12,000 transactions / minute The result showed the high reliability and availability of

GlassFish

Improvement to handle a huge traffic

Resolve issues & challenge to upper goalSome issues

Instance down due to the full of task queue Unknown exception on server.log

Challenge to upper goal

org.glassfish.flashlight.impl.client.ReflectiveClientInvokerjava.lang.reflect.InvocationTargetExceptionCAUSE:  java.lang.NullPointerExceptionid=101target=org.glassfish.web.admin.monitor.HttpServiceStatsProvider@193e1fcmethod=public void org.glassfish.web.admin.monitor.HttpServiceStatsProvider.connectionAcceptedEvent(java.lang.String,int,java.lang.String)paramNames=[listenerName, connection, address]probeIndices=[0, 1, 2]useProbeArgs=truehasComputedParams=false