Upload
aragozin
View
3.931
Download
1
Embed Size (px)
DESCRIPTION
Slide deck from JUG at Saint-Petersburg 12 Dec 2013
Citation preview
Virtualizing Java in Java
Alexey Ragozin
How?
jug.ru 12 Dec 2013
Virtualizing Java in Java
Single JVM hosting • multiple “pseudo JVMs”, which
• Have independent system properties
• Have independent statics
• May have different classpath
• May be forcefully terminated or suspended/resumed
You can deploy whole distributed application topology inside of JUnit test
All started with Oracle Coherence
Native Java distributed peer-to-peer key/value storage
With a lot of extension points
Distributed data processing
Listeners for remote events
Read-through / write-through patterns
Pluggable serialization
And inability to start two cluster members in one JVM
And we need to test
Behavior of very specific features
Serialization/deserialization defects
Data routing and collocation aspects
Code meant for distributed execution
Threading aspects
Classpath differences between cluster processes
Cluster configuration tweaks
Multiplexing singletons
Custom classloader
Force to reload classes already loaded by parent (second copy of class loaded)
Got our first cluster-in-JVM alive
InheritentThreadLocal to add fancy stuff
– Multiplexing system properties
– Multiplexing multiplex console output
Heated competition
Three open source “cluster virtualizing” effort around Oracle Coherence
GridKit (ChTest) code.google.com/p/gridkit
Oracle tools github.com/coherence-community/oracle-tools
Little www.littlegrid.net
“Distributing” test case
How to start The Application on “virtual cluster”?
Old school
Main classes and command line arguments
But if you need to do verification inside of vinode?
A separate main for each test case?
But how all this is relevant to me?
Normal client/server application You can use your real main classes instead of mocking server
Hadoop / HBase / Cassandra Distributed
Deployment unfriendly
Ship with single node – cut down versions
“Distributing” test case
Transparent remotting
client1.exec(new Runnable() {
@Override
public void run() {
NamedCache cache = CacheFactory.getCache(cacheName);
Assert.assertNull(cache.get("A"));
cache.put("A", "aaa");
}
});
client2.exec(new Runnable() {
@Override
public void run() {
NamedCache cache = CacheFactory.getCache(cacheName);
Assert.assertEquals("aaa", cache.get("A"));
}
});
We want to test more!
We are in control class loading
— let’s tweak classpath on flight
Inject resources
Remove server classes from client
Test different codebase versions
Master JVM, client regression test pack [trunk version]
Backward compatibility testing
Client [version X]
Server [trunk version]
Case
Cross version tests
Master JVM, client regression test pack [trunk version]
Client [version X]
Server [version Z]
Client [version Y]
Case
Managing artifacts
… a bunch of black magic to find local repo
and managing classpath as easy as …
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.8</version>
<executions>
<execution>
<id>viconcurrent-0.7.15</id>
<phase>test-compile</phase>
<goals>
<goal>get</goal>
</goals>
<configuration>
<artifact>org.gridkit.lab:viconcurrent:0.7.15</artifact>
</configuration>
</execution>
</executions>
</plugin>
Managing artifacts
How to get needed artifact on local disk
- Maven will disallow two versions of same artifact
- but we can trick it …
Transitive dependencies are not included, though.
ViNode node;
…
node.x(MAVEN).replace("org.gridkit.lab", "viconcurrent", "0.7.15");
Transparent remotting
Own implementation of RMI
Standard Java serialization
Serializing of anonymous Runnable/Callable
Auto export of Remote interfaces
Writing distributed code by convention!
Bidirectional communications
public interface RemotePut extends Remote { public void put(Object key, Object value); }
RemotePut remoteService = client1.exec(new Callable<RemotePut>() { @Override public RemotePut call() { final NamedCache cache = CacheFactory.getCache(cacheName); return new RemotePut() { @Override public void put(Object key, Object value) { cache.put(key, value); } }; } });
public interface RemotePut extends Remote { public void put(Object key, Object value); } @SuppressWarnings("unused") @Test public void bidirectional_remoting() { // Present for typical single node cluster cloud.all().presetFastLocalCluster(); cloud.node("storage.**").localStorage(true); cloud.node("client.**").localStorage(false); // Simulates DefaultCacheServer based process cloud.node("storage.**").autoStartServices(); // declaring specific nodes to be created CohNode storage = cloud.node("storage.1"); CohNode client1 = cloud.node("client.1"); CohNode client2 = cloud.node("client.2"); // now we have 3 specific nodes in cloud // all of then will be initialized in parallel cloud.all().ensureCluster(); final String cacheName = "distr-a"; RemotePut remoteService = client1.exec(new Callable<RemotePut>() { @Override public RemotePut call() { final NamedCache cache = CacheFactory.getCache(cacheName); return new RemotePut() { @Override public void put(Object key, Object value) { cache.put(key, value); } }; } }); remoteService.put("A", "aaa"); client2.exec(new Runnable() { @Override public void run() { NamedCache cache = CacheFactory.getCache(cacheName); Assert.assertEquals("aaa", cache.get("A")); } }); }
remoteService.put("A", "aaa");
Bidirectional communications
public interface RemotePut extends Remote { public void put(Object key, Object value); }
RemotePut remoteService = client1.exec(new Callable<RemotePut>() { @Override public RemotePut call() { final NamedCache cache = CacheFactory.getCache(cacheName); return new RemotePut() { @Override public void put(Object key, Object value) { cache.put(key, value); } }; } });
remoteService.put("A", "aaa");
Extending java.rmi.Remotexec will mark interface for auto export
Unlike Java RMI, there is no need to declare RemoteException for every method
Result of callable will be serialized and transferred to caller
Objects implementing remote interfaces are automatically replaced with remote stub during serialization
Here we got a remote stub, not a real implementation of interface
Call to a stub, will be converted to “remote” call to instance we have created in “virtualized” node few lines above
Sneak peek: Instrumentation
System.exit() – is still fatal
Some cases need “virtual time”
Tweaking monolithic code
Fault injection
Mock injection
Sneak peek: Instrumentation
PowerMock
Recompiles everything (Coherence ~ 5000 classes)
AspectJ
Static interceptors
ByteMan
Using agent + weird language
Sneak peek: Instrumentation
ViNode node = ...
ViHookBuilder
.newCallSiteHook()
.onTypes(System.class)
.onMethod("exit")
.doReturn(null)
.apply(node);
node.exec(new Callable<Void>() {
@Override
public Void call() throws Exception {
System.exit(0);
return null;
}
});
Questions?
LET’S TAKE A BREAK HERE
From “virtual” to real cluster
Alexey Ragozin
jug.ru 12 Dec 2013
Virtual stuff is so good
Managing virtual nodes in deterministic way, in Java, having all luxury of exception handling and richness of libraries – feeling were so good …
I wished, I could rollout JVMs across real servers
Your network is Big JVM
Same API
3 types of nodes: in-process, local, remote
Transparent remotting
SSH to manage remote server
Automatic classpath replication (with caching)
Zero infrastructure
Any OS for master host
SSHd + JVM for slave hosts
New opportunities
Performance testing deploy system under test
deploy load generators
deploy monitoring agents
gather all result in one place
Deployment (remote execution task for ANT)
Replace your putty with Java IDE log scrapping
parallel execution
As easy as …
@Test
public void remote_hello_world() throws InterruptedException
{
ViManager cloud = CloudFactory.createSimpleSshCloud();
cloud.node("myserver.uk.db.com");
cloud.node("**").exec(new Callable<Void>() {
@Override
public Void call() throws Exception
{
String localHost = InetAddress.getLocalHost().toString();
System.out.println("Hi! I'm running on " + localHost);
return null;
}
});
}
Behind the scene
• JSCh – SSH client (slightly patched)
• Collected classpath artifacts
• SCP jars to remote target
• Start remote agent (JVM)
stdOut / stdIn for master – agent communications
• Agent start slave process
• Slave start RMI node and connects (TCP) to master
agent acts as TCP proxy
Death clock is ticking
Master JVM kills slave processes, unless
SSH session was interrupted
someone kill -9 master JVM
master JVM has crashed (e.g. under debuger)
Death clock is ticking on slave though
if master is not responding
slave process will terminate itself
Performance testing
• Master JVM is running on CI
• JUnit to start test
• Loaded generator farm is deployed on test servers
• Monitoring agents deployed on application cluster
• Metrics are buffered locally, then send to master and processed
Real test
• Four servers – application
• 50 servers – load farm
• Over 200 – slaves processes
Coding for 200+ processes
Driver - concept
• Driver – Java interface encapsulates test action
• One way methods
• Friendly for remotting for parallel invokation
+ some utility for parallel execution, workflow etc
Example:
https://gridkit.googlecode.com/svn/grid-lab/trunk/examples/zk-benchmark-sample
Monitoring
Sigar - http://www.hyperic.com/products/sigar
• collecting common system metrics
Attach API – built-n JVM monitoring
SJK – spin-off CLI tool for JVM diagnostics
• https://github.com/aragozin/jvm-tools
Links
NanoCloud • https://code.google.com/p/gridkit/wiki/NanoCloudTutorial
• Maven Central: org.gridkit.lab:telecontrol-ssh:0.7.22
• http://blog.ragozin.info/2013/01/remote-code-execution-in-java-made.html
ANT task • https://github.com/gridkit/gridant
ChTest (Coherence test tool) • https://code.google.com/p/gridkit/wiki/ChTest
• Maven Central: org.gridkit.coherence-tools:chtest:0.2.4
Thank you
Alexey Ragozin [email protected]
http://blog.ragozin.info - my articles http://code.google.com/p/gridkit http://github.com/gridkit - my open source code http://aragozin.timepad.ru - tech meetups in Moscow