Cloud Computing Lec7 - Kursused · PDF fileBasics of Cloud Computing –Lecture 7 More AWS and Cloud-based ... Oracle or Microsoft SQL Server • NoSQLdatabases ... • Cisco believes

Basics of Cloud Computing – Lecture 7

More AWS and Cloud-based Research at Mobile & Cloud Lab

Satish Srirama

Outline

• More Amazon Web Services

• How we are using cloud

• Cloud based Research @ Mobile & Cloud Lab

3/21/2017 Satish Srirama 2/41

Cloud Providers and Services – we already discussed

• Amazon Web Services– Amazon EC2– Amazon S3– Amazon EBS– Amazon Elastic Load Balancing– Amazon Auto Scale– Amazon CloudWatch

• Eucalyptus• OpenStack• SciCloud• Management providers

– ElasticFox– RightScale

• PaaS– Google AppEngine– Windows Azure


MORE AWS

3/21/2017 Satish Srirama 4

AWS we discuss

• AWS Management Console

• AWS Identity and Access Management

• AWS CloudFormation

• Amazon Elastic MapReduce


AWS Management Console

• Hope some of you have started using Amazon accounts

• You can manage your complete Amazon account with management console (Similar to Hybridfox)– AMI Management– Instance Management– Security Group Management– Elastic IP Management– Elastic Block Store– Key Pair management etc.

• Have different panes for different services


AWS Management Console - screenshot

https://console.aws.amazon.com/


AWS Identity and Access Management (IAM)

• How can an enterprise or group of people use a single credit card?

• Manage IAM users– Create new users and manage them

– Create groups

• Manage permissions– Creating policies

• Manage credentials– Create and assign temporary security credentials


IAM policy

• Example policy giving access to complete EC2

http://aws.amazon.com/iam/


AWS CloudFormation

• Provides an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion

• It is based on templates model– Templates describe the AWS resources, the associated

dependencies, and runtime parameters to run an app. – The templates describe stacks, which are set of software and

hardware resources. – Something similar to CloudML and RightScale server templates

• Hides several details– How the AWS services need to be provisioned– Subtleties of how to make those dependencies work.


AWS CloudFormation• Amazon provides several pre-built templates to

start common apps as:– WordPress (blog)

– LAMP stack

– Gollum (wiki used by GitHub)

– …

• There is no additional charge for AWS CloudFormation. You pay for AWS resources (e.g. EC2 instances, Elastic Load Balancers, etc.) http://aws.amazon.com/cloudformation/


Amazon Elastic MapReduce

• Web interface and command-line tools for running Hadoop jobs on EC2

• Data stored in Amazon S3

• Monitors job and shuts machines after use

• Running a job– Upload job jar & input data to S3

– Create the cluster

– Create a Job Flow as steps

– Wait for the completion and examine the results

3/21/2017

http://aws.amazon.com/elasticmapreduce/

Satish Srirama 12/41

Other interesting AWS

• Amazon Relational Database Service

– Provides access to the capabilities of familiar database engines

– MySQL, Oracle or Microsoft SQL Server

• NoSQL databases

– Simple DB

– DynamoDB


CLOUD BASED RESEARCH @

MOBILE & CLOUD LAB


Scientific Computing on the Cloud

• Public clouds provide very convenient access to computing resources– On-demand and in real-time– As long as you can afford them

• High performance computing (HPC) on cloud– Virtualization and communication latencies are major

hindrances [Srirama et al, SPJ 2011; Batrashev et al, HPCS 2011]

• Things have improved significantly over the years• Containerization is a good alternative

– Research at scale• Cost-to-value of experiments


Communication pattern of Cluster vs Cloud

• Cloud has huge troubles with communication/transmission latencies– Virtualization technology is the culprit

• Performance Comparison of virtual machines and Linux containers (e.g. Docker)

3/21/2017 Satish Srirama16/31

[Srirama et al, SPJ 2011]

Adapting Computing Problems to Cloud

• Reducing the algorithms to cloud computing frameworks like MapReduce [Srirama et al, FGCS 2012]

• Designed a classification on how the algorithms can be adapted to MR– Algorithm � single MapReduce job

• Monte Carlo, RSA breaking

– Algorithm � n MapReduce jobs• CLARA (Clustering), Matrix Multiplication

– Each iteration in algorithm � single MapReduce job• PAM (Clustering)

– Each iteration in algorithm � n MapReduce jobs • Conjugate Gradient

• Applicable especially for Hadoop MapReduce


Issues with Hadoop MapReduce

• It is designed and suitable for:– Data processing tasks

– Embarrassingly parallel tasks

• Has serious issues with iterative algorithms– Long „start up“ and „clean up“ times ~17 seconds

– No way to keep important data in memory between MapReduce job executions

– At each iteration, all data is read again from HDFS and written back there at the end

– Results in a significant overhead in every iteration


Alternative Approaches

• Restructuring algorithms into non-iterative versions – CLARA instead of PAM [Jakovits & Srirama, Nordicloud 2013]

• Alternative MapReduce implementations that are designed to handle iterative algorithms [Jakovits and Srirama, HPCS 2014]

– E.g. Twister, HaLoop, Spark [Large-scale Data Processing on the Cloud - MTAT.08.036

(Fall 2017)]

• Alternative distributed computing models– Bulk Synchronous Parallel model [Valiant, 1990] [Jakovits et al, HPCS

2013]

– Building a fault-tolerant BSP framework (NEWT) [Kromonov et

al, HPCS 2014]


3/21/2017

[Tomi T Ahonen]

Satish Srirama 20

Mobile Applications

• One can do interesting things on mobiles directly– Today’s mobiles are far more capable

– Location-based services (LBSs), mobile social networking, mobile commerce, context-aware services etc.

• It is also possible to make the mobile a service provider– Mobile web service provisioning [Srirama et al, ICIW 2006; Srirama

and Paniagua, MS 2013]

– Challenges in security, scalability, discovery and middleware are studied [Srirama, PhD 2008]

– Mobile Social Network in Proximity [Chang et al, ICSOC 2012; PMC 2014]


However, we still have not achieved

• Longer battery life– Battery lasts only for 1-2 hours for continuous

computing

• Same quality of experience as on desktops– Weaker CPU and memory

– Storage capacity

• Still it is a good idea to take the support of external resources for building resource intensive mobile applications


Mobile Cloud ApplicationsMobile Application Development - MTAT.03.262 (Fall 2017)

• Bring the cloud infrastructure to the proximity of the mobile user

• Mobile has significant advantage by going cloud-aware

– Increased data storage capacity

– Availability of unlimited processing power

– PC-like functionality for mobile applications

– Extended battery life (energy efficiency)


Mobile Cloud – Our interpretation

• We do not see Mobile Cloud to be just a scenario where mobile is taking the help of a much powerful machine!!!

• We do not see cloud as just a pool of virtual machines

• Mobile Cloud based system should take advantage of some of the key intrinsic characteristics of cloud efficiently– Elasticity & AutoScaling

– Utility computing models

– Parallelization (e.g., using MapReduce)


Mobile Cloud Binding Models

3/21/2017

Task Delegation Code Offloading

[Flores & Srirama, JSS 2014]

[Flores et al, MoMM 2011]

Mobile Cloud

[Flores and Srirama, MCS 2013]


MCM – enables

• Interoperability between different Cloud Services (IaaS, SaaS, PaaS) and Providers (Amazon, Eucalyptus, etc)

• Provides an abstraction layer on top of API

• Composition of different Cloud Services

• Asynchronous communication between the device and MCM [Warren et al, IEEE PC 2014]

• Means to parallelize the tasks and take advantage of Cloud’s intrinsic characteristics

3/21/2017

[Flores et al, MoMM 2011]


MCM applications

• CroudSTag [Srirama et al, MobiWIS 2011]

– Social group formation with people identified in Pictures/Videos

• Zompopo [Srirama et al, NGMAST 2011]

– Intelligent calendar, by mining accelerometer sensor data

• Bakabs [Paniagua et al, iiWAS-2011]

– Managing the Cloud resources from mobile

• Sensor data analysis– Human activity recognition – Context aware gaming– MapReduce based sensor data analysis [Paniagua et al, MobiWIS 2012]

• SPiCa: A Social Private Cloud Computing Application Framework [Chang et al, MUM 2014]


Code Offloading - Major Components

• Major research challenges – What, when,

where and how to offload?

• Mobile– Code profiler– System profilers– Decision engine

• Cloud based surrogate platform

3/21/2017

[Flores and Srirama, MCS 2013]


Practical adaptability of offloading

Applications that can benefit became limited with increase in device capacities 29/41

[Srirama, TICT 2017]

Internet of Things (IoT)

• “The Internet of Things allows people and

things to be connected Anytime, Anyplace,

with Anything and Anyone, ideally using Any

path/network and Any service.” [European

Research Cluster on IoT]

• More connected devices than people

• Cisco believes the market size will be $19 trillion by 2025

3/21/2017 Satish Srirama30

IoT - Scenarios

• Environment Protection

• Smart Home

• Smart Healthcare

• Smart Agriculture

[Kip Compton][Perera et al, TETT 2014]

3/21/2017 Satish Srirama31[http://www.libelium.com/improving-banana-crops-production-and-agricultural-sustainability-in-colombia-using-sensor-networks/ ]

Internet of Things – Challenges

Sensors Tags Mobile Things

Appliances & Facilities

How to interactwith ‘things’

directly?

How to provide energy efficient

services?

How do we communicate automatically?

[Chang et al, ICWS 2015]

[Chang et al, SCC 2015; Liyanage et al, MS 2015]


Cloud-based IoT

Sensing and smart devices

Connectivity nodes &Embedded processing

Remote Cloud-based processing

Proxy Storage

Processing


Research focus in IoT

• We have established IoT and Smart Solutions Lab with Telia company support

• Interesting topics

– Discovery of IoT devices

– Working with IoT based devices

– Study of available IoT platforms

• Amazon IoT

• Open IoT

3/21/2017 34 Satish Srirama

IoT Data Processing on Cloud

• Enormous amounts of unstructured data– In Zetabytes (1021 bytes) by 2020 [TelecomEngine]

– Has to be properly stored, analysed and interpreted and presented

• Big data acquisition and analytics– Is MapReduce sufficient?

• MapReduce is not good for iterative algorithms [Srirama et al, FGCS 2012]

– IoT mostly deals with streaming data• Message queues such as Apache Kafka can be used to buffer and feed

the data into stream processing systems such as Apache Storm• Apache Spark streaming

• How to ensure QoS aspects such as security of data?– Anonymization and Expiry of data?

• Especially for the personal data


WE ALWAYS WELCOME NEW IDEAS!

email: [email protected]


This week in lab

• Continue working with Google AppEngine


Next week - Examination

• Exam I : Monday, 27.03.2017, 10:00 – 12:00, Ülikooli 17 /Paabel/ room 220

• Exam II : Tuesday, 28.03.2017, 10:00 – 12:00 in room J. Liivi 2 - 402


Preparation for Examination

• One of the earlier exam paper is kept online

• Slides are mostly self sufficient

• References are mentioned for further reading

• Mainly focus at keywords


References

• Check Amazon videos and webinars at http://aws.amazon.com/resources/webinars/• List of Publications - Satish Narayana Srirama - http://math.ut.ee/~srirama/publications.html• [Flores et al, IEEE Communications Mag 2015] H. Flores, P. Hui, S. Tarkoma, Y. Li, S. N. Srirama, R. Buyya: Mobile Code Offloading: From Concept to

Practice and Beyond, IEEE Communications Magazine, ISSN: 0163-6804, 53(3):80-88, 2015. IEEE. DOI:10.1109/MCOM.2015.7060486• [Chang et al, SCC 2015]C. Chang, S. N. Srirama, J. Mass: A Middleware for Discovering Proximity-based Service-Oriented Industrial Internet of Things,

12th IEEE International Conference on Services Computing (SCC 2015), June 27 - July 2, 2015, pp. 130-137. IEEE.• [Liyanage et al, MS 2015] M. Liyanage, C. Chang, S. N. Srirama: Lightweight Mobile Web Service Provisioning for Sensor Mediation, 4th International

Conference on Mobile Services (MS 2015), June 27 - July 2, 2015, pp. 57-64. IEEE• [Chang et al, ICWS 2015] C. Chang, S. Loke, H. Dong, F. Salim, S. N. Srirama, M. Liyanage, S. Ling: An Energy-Efficient Inter-organizational Wireless

Sensor Data Collection Framework, The IEEE 22nd International Conference on Web Services (ICWS 2015), June 27 - July 2, 2015, pp. 639-646. IEEE.• [Flores and Srirama, JSS 2014] H. Flores, S. N. Srirama: Mobile Cloud Middleware, Journal of Systems and Software, ISSN: 0164-1212, 92(1):82-94,

2014. Elsevier. DOI: 10.1016/j.jss.2013.09.012.• [Chang et al, PMC 2014] C. Chang, S. N. Srirama, S. Ling: Towards an Adaptive Mediation Framework for Mobile Social Network in Proximity,

Pervasive and Mobile Computing Journal, MUCS Fast track, ISSN: 1574-1192, 12:179-196, 2014. Elsevier. DOI: 10.1016/j.pmcj.2013.02.004.• [Warren et al, IEEE PC 2014] I. Warren, A. Meads, S. N. Srirama, T. Weerasinghe, C. Paniagua: Push Notification Mechanisms for Pervasive

Smartphone Applications, IEEE Pervasive Computing, ISSN: 1536-1268, 13(2):61-71, 2014. IEEE. DOI:10.1109/MPRV.2014.34. • [Chang et al, MUM 2014] C. Chang, S. N. Srirama, S. Ling: SPiCa: A Social Private Cloud Computing Application Framework, The 13th International

Conference on Mobile and Ubiquitous Multimedia (MUM 2014), November 25-28, 2014, pp. 30-39. ACM. • [Jakovits and Srirama, HPCS 2014] P. Jakovits, S. N. Srirama: Evaluating MapReduce Frameworks for Iterative Scientific Computing Applications, The

2014 (12th) International Conference on High Performance Computing & Simulation (HPCS 2014), July 21-25, 2014, pp. 226-233. IEEE.• [Kromonov et al, HPCS 2014] I. Kromonov, P. Jakovits, S. N. Srirama: NEWT - A resilient BSP framework for iterative algorithms on Hadoop YARN, The

2014 (12th) International Conference on High Performance Computing & Simulation (HPCS 2014), July 21-25, 2014, pp. 251-259. IEEE. • [Srirama and Viil, HPCC 2014] S. N. Srirama, J. Viil: Migrating Scientific Workflows to the Cloud: Through Graph-partitioning, Scheduling and Peer-to-

Peer Data Sharing, 16th Int. Conf. on High Performance and Communications (HPCC 2014) workshops, August 20-22, 2014, pp. 1137-1144. IEEE. • [Jakovits and Srirama, Nordicloud 2013] P. Jakovits, S. N. Srirama: Clustering on the Cloud: Reducing CLARA to MapReduce, 2nd Nordic Symposium

on Cloud Computing & Internet Technologies (NordiCloud 2013), September 02-03, 2013, pp. 64-71. ACM. • [Jakovits et al, HPCS 2013] P. Jakovits, S. N. Srirama, I. Kromonov: Viability of the Bulk Synchronous Parallel Model for Science on Cloud, The 2013

(11th) International Conference on High Performance Computing & Simulation (HPCS 2013), July 01-05, 2013, pp. 41-48. IEEE. • [Srirama and Paniagua, MS 2013] S. N. Srirama, C. Paniagua: Mobile Web Service Provisioning and Discovery in Android Days, The 2013 IEEE

International Conference on Mobile Services (MS 2013), June 27 - July 02, 2013, pp. 15-22. IEEE.• [Flores and Srirama, MCS 2013] H. Flores, S. N. Srirama: Adaptive Code Offloading for Mobile Cloud Applications: Exploiting Fuzzy Sets and Evidence-

based Learning, The Fourth ACM Workshop on Mobile Cloud Computing and Services (MCS 2013) @ MobiSys 2013, June 25-28, 2013, pp. 9-16. ACM.• [Oliner et al, 2013] Oliner, Adam J., Anand P. Iyer, Ion Stoica, Eemil Lagerspetz, and Sasu Tarkoma. "Carat: Collaborative energy diagnosis for mobile

devices." In Proceedings of the 11th ACM Conference on Embedded Networked Sensor Systems, p. 10. ACM, 2013.• [Srirama et al, SOCA 2012] S. N. Srirama, C. Paniagua, H. Flores: Social Group Formation with Mobile Cloud Services, Service Oriented Computing and

Applications Journal, ISSN: 1863-2386, 6(4):351-362, 2012. Springer. DOI: 10.1007/s11761-012-0111-5.• [Srirama et al, FGCS 2012] S. N. Srirama, P. Jakovits, E. Vainikko: Adapting Scientific Computing Problems to Clouds using MapReduce, Future

Generation Computer Systems Journal, 28(1):184-192, 2012. Elsevier press. DOI 10.1016/j.future.2011.05.025.


References - continued• [Chang et al, ICSOC 2012] C. Chang, S. N. Srirama, S. Ling: An Adaptive Mediation Framework for Mobile P2P Social

Content Sharing, 10th International Conference on Service Oriented Computing (ICSOC 2012), November 12-16, 2012, pp. 374-388. Springer LNCS.

• [Paniagua et al, MobiWIS 2012] C. Paniagua, H. Flores, S. N. Srirama: Mobile Sensor Data Classification for Human Activity Recognition using MapReduce on Cloud, The 9th Int. Conf. on Mobile Web Information Systems (MobiWIS2012), August 27-29, 2012, pp. 585-592. Elsevier.

• [Srirama et al, SPJ 2011] S. N. Srirama, O. Batrashev, P. Jakovits, E. Vainikko: Scalability of Parallel Scientific Applications on the Cloud, Scientific Programming Journal, Special Issue on Science-driven Cloud Computing, 19(2-3):91-105, 2011. IOS Press. DOI 10.3233/SPR-2011-0320.

• [Flores et al, MoMM 2011] H. Flores, S. N. Srirama, C. Paniagua: A Generic Middleware Framework for Handling Process Intensive Hybrid Cloud Services from Mobiles, The 9th International Conference on Advances in Mobile Computing & Multimedia (MoMM-2011), December 5-7, 2011, pp. 87-95. ACM.

• [Paniagua et al, iiWAS 2011] C. Paniagua, S. N. Srirama, H. Flores: Bakabs: Managing Load of Cloud-based Web Applications from Mobiles, The 13th International Conference on Information Integration and Web-based Applications & Services (iiWAS-2011), December 5-7, 2011, pp. 489-495. ACM.

• [Srirama et al, MobiWIS 2011] S. N. Srirama, C. Paniagua, H. Flores: CroudSTag: Social Group Formation with Facial Recognition and Mobile Cloud Services, The 8th International Conference on Mobile Web Information Systems (MobiWIS 2011), September 19-21, 2011, v. 5 of Procedia Computer Science, pp. 633-640. Elsevier. doi: 10.1016/j.procs.2011.07.082.

• [Srirama et al, NGMAST 2011] S. N. Srirama, H. Flores, C. Paniagua: Zompopo: Mobile Calendar Prediction based on Human Activities Recognition using the Accelerometer and Cloud Services, 5th International Conference on Next Generation Mobile Applications, Services and Technologies (NGMAST 2011), September 14-16, 2011, pp. 63-69. IEEE.

• [Batrashev et al, HPCS 2011] O. Batrashev, S. N. Srirama, E. Vainikko: Benchmarking DOUG on the Cloud, The 2011 International Conference on High Performance Computing & Simulation (HPCS 2011), July 4-8, 2011, pp. 677-685. IEEE.

• [Srirama, PhD 2008] S. N. Srirama: Mobile Hosts in Enterprise Service Integration, PhD thesis, RWTH Aachen University, September, 2008.

• [Srirama et al, ICIW 2006] S. N. Srirama, M. Jarke, W. Prinz: Mobile Web Service Provisioning, Proceedings of the Advanced International Conference on Telecommunications and International Conference on Internet and Web Applications and Services (AICT-ICIW 2006), February 23-25, 2006, pp. 120-125. IEEE Computer Society Press.

• [Valiant, 1990] L. G. Valiant: A bridging model for parallel computation, Commun. ACM, vol. 33, no. 8, pp. 103–111, Aug. 1990.


Documents

Cloud Computing Lec7 - Kursused · PDF fileBasics of Cloud Computing –Lecture 7 More AWS and Cloud-based ... Oracle or Microsoft SQL Server • NoSQLdatabases ... • Cisco believes