Upload
takayuki-shimizu
View
13
Download
4
Embed Size (px)
DESCRIPTION
Citation preview
Shoot the Frieza with Amazon Kinesis !
JAWSDAYS 2014.3.15 @shimy_net
I’m AWS Comedian Takayuki SHIMIZU @shimy_net
AWS Comedian have to waste a technical talent for getting a laugh.
My works
Cycling with DynamoDB AWS Automation by
Rube Goldberg machine
EC2 Kickstart by Kinect
Today’s theme
Kinesis
What’s Amazon Kinesis ?
Kinesis
Big data Real-time Processing Visualize
We can visualize everything in real-time.
In short
Got an idea !
If we put the POWER into Amazon Kinesis …
We can defeat Frieza
… I can annihilate you with only half of my full
strength.
Imagine…
Collect the POWER JUST
GIVE ME A LITTLE BIT OF YOUR
POWER..
EARTH…
OCEAN…
ALL LIVING
THINGS…
Put it into Amazon Kinesis
Visualize it !
We can defeat Frieza
(゚ー゚*;) OK
(́Д`;) Really OK?
How to collect the POWER ?
Tweet Morphological Analysis Emotion Analysis Estimate the POWER
1Tweet = Max 100 pt Convert Tweet into POWER
POWER
Step1. Morphological Analysis
本日は晴天なり
本日 / は / 晴天 / なり
MeCab It's fine today
It / is / fine / today
Step2. Emotion Analysis
本日 / は / 晴天 / なり
Emotion Dictionary -1.0~+1.0 (Negative or Positive)
+0.63 -0.18
fine today
Step3. Estimate the POWER
POWER = 72 pt
本日 / は / 晴天 / なり +0.63 -0.18
fine today
Step4. Create an Energy ball
82pt
37pt 77pt
64pt
93pt
Tweet
Tweet
Tweet
Tweet
Tweet
Break the limit over 530,000pt !!!
Don’t worry. I won’t use all of my power.
It’s 530,000
I wrote a program
Architecture
Kinesis Stream
Twitter4J Kinesis MeCab Dynamo Unity
Visualize Analyze Kinesis Tweet
We fight with #jawsdays’s POWER
Put tweets into Kinesis Kinesis Stream
Twitter4J Kinesis MeCab Dynamo Unity
TwitterStream twitterStream = new TwitterStreamFactory().getInstance();StatusListener listener = new StatusListener() { @Override public void onStatus(Status status) {
String key = RandomStringUtils.randomAlphanumeric(10);String json = DataObjectFactory.getRawJSON(status); PutRecordRequest putRecordRequest = new PutRecordRequest();putRecordRequest.setStreamName(PropUtil.getInstance().getProp("StreamName"));putRecordRequest.setData(ByteBuffer.wrap(json.getBytes(Charset.forName("UTF-8")));putRecordRequest.setPartitionKey(key);PutRecordResult putRecordResult = kinesis.putRecord(putRecordRequest);
}};twitterStream.addListener(listener);FilterQuery filterQuery = new FilterQuery();filterQuery.track(new String[] { PropUtil.getInstance().getProp("Keyword1"), PropUtil.getInstance().getProp("Keyword2") });twitterStream.filter(filterQuery);
Get tweets from Kinesis Kinesis Stream
Twitter4J Kinesis MeCab Dynamo Unity
private void processRecordsWithRetries(List<Record> records) { for (Record record : records) { boolean processedSuccessfully = false; String data = null; for (int i = 0; i < NUM_RETRIES; i++) { try { // Get Data data = decoder.decode(record.getData()).toString(); Status status = DataObjectFactory.createStatus(data ); // MeCab something // ・・・・・・
processedSuccessfully = true; break; } catch (Throwable t) {} }}
}
Analyze tweets Kinesis Stream
Twitter4J Kinesis MeCab Dynamo Unity Tagger tagger = new Tagger("-Ochasen");tagger.parse(status.getText().replaceAll("¥n", ""));Node node = tagger.parseToNode(status.getText().replaceAll("¥n", ""));Double sum = 0.0;int count = 0;for (; node != null; node = node.getNext()) {
String[] mecabArray = node.getFeature().split(","); String originalWord = node.getSurface();if (!mecabArray[6].equals("*")) { originalWord = mecabArray[6]; // 基本形 } String feature = mecabArray[0];if (feature.equals("名詞") || feature.equals("動詞") || feature.equals("形容詞”) || feature.equals("形容動詞") || feature.equals("副詞")) { // Positive or Negative sum += PositiveNegativeJudge.getInstance().getDouble(originalWord + ":" + feature); count++;}
}Double point = Math.floor((sum / count + 1.0) * 100.0 / 2.0);
Store results on Dynamo DB Kinesis Stream
Twitter4J Kinesis MeCab Dynamo Unity
private static PutItemRequest createPutItemRequest(String id, String datetime, String name, String screenName, String text, Double value) {
Map<String, AttributeValue> putItems = new HashMap<String, AttributeValue>();putItems.put("AppName", new
AttributeValue().withS(PropUtil.getInstance().getProp("ApplicationName")));putItems.put("Datetime/SerialNumber", new AttributeValue().withS(datetime+"/"+id));putItems.put("ScreenName", new AttributeValue().withS("@" + screenName));putItems.put("Name", new AttributeValue().withS(name));putItems.put("Text", new AttributeValue().withS(text));putItems.put("Value", new AttributeValue().withN(Double.toString(value)));PutItemRequest putItemRequest = new
PutItemRequest().withTableName(PropUtil.getInstance().getProp("DynamoDbTableNameDetail")).withItem(putItems);
return putItemRequest;}
Store results on Dynamo DB
The POWER
Total POWER Atomic Counter
Output to JSON
Visualize the Power using Unity Kinesis Stream
Twitter4J Kinesis MeCab Dynamo Unity
Hatsune Miku ※Virtual idol
Here we go !
Give me the POWER
In real-time Miku collects the POWER
Miku
Everyone’s tweet
POWER meter
Kamehameha
Frieza
Demonstration
Lose
We fight with #jawsdays’s POWER
Miku falls down
Lose
We need more POWER
We need more POWER !!!
This situation is within my “assumption”
if (tweet.indexOf(“芸人イケメン”)) != -1) {
point = 530000.0;
}
”Comedian Cool”
Give me your POWER !
芸人イケメン #jawsdays
Please tweet “Comedian Cool” and give me your POWER !!
Comedian Cool
Thanks for a lot of tweets
Retry
Kamehameha
Win
Over 530,000 !!!
Frieza
Frieza
Win
We did it !
Comedian Cool Kinesis Cool
Conclusion
Inside story I monitored processes of my program in server-side because I had to accomplish this real-time demonstration within 5 min. But an incident happened. Just before my turn to go on stage, I noticed that a process which retrieve POWER’s data from Dynamo DB was often stopped. ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter ec2-user 18622 2.2 4.8 1787796 82660 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter ec2-user 18944 22.0 4.2 1618412 71836 ? Sl 06:16 0:08 java DynamoJsonCreator ↓ ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter ec2-user 18622 2.2 4.9 1839496 83452 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter
Inside story I tried to set an alive-monitoring by “cron” and be able to restart a process automatically when it’s dead. crontab -l */2 * * * * bash /home/ec2-user/KinesisTwitter/check_putter.sh > /home/ec2-user/cron.log.putter.txt 2>&1 */2 * * * * bash /home/ec2-user/KinesisTwitter/check_getter.sh > /home/ec2-user/cron.log.getter.txt 2>&1 */2 * * * * bash /home/ec2-user/KinesisTwitter/check_create_json.sh > /home/ec2-user/cron.log.create.json.txt 2>&1
In the meantime, I found my mistake that Dynamo DB’s throughput set 1. (;´Д`) oh…
Inside story Nintendo’s presentation had started and the frequency of #jawsdays’s tweet was increasing. Then my program’s process was stopped. I modified Dynamo DB’s throughput to solve a problem. This is the quintessence of AWS.
Nintendo’s presentation start
Modify throughput
References ドラゴンボール Amazon Kinesis http://aws.amazon.com/jp/kinesis/ Amazon DynamoDB http://aws.amazon.com/jp/dynamodb/ Twitter4J http://twitter4j.org/ja/ MeCab http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html MeCab-java 単語感情極性対応表 http://www.lr.pi.titech.ac.jp/~takamura/pndic_ja.html Unity http://japan.unity3d.com/ MikuMikuDcance for Unity http://mmd-for-unity-proj.github.io/mmd-for-unity/ Lat式ミク http://dic.nicovideo.jp/a/lat フリーザ http://ux.getuploader.com/karota318/ かめはめ波モーション http://www.nicovideo.jp/watch/sm15093547 歩き・スキップモーション http://www.nicovideo.jp/watch/sm21263509 待機モーション http://www.nicovideo.jp/watch/sm18961728 やられモーション http://www.nicovideo.jp/watch/sm19073965 日常モーション http://www.nicovideo.jp/watch/sm18015670
Thank you
Well,
goodbye !