Upload
sheila
View
38
Download
0
Tags:
Embed Size (px)
DESCRIPTION
A.R.M.S. Active Resource Management Services For Big Data Processing. Revised Presentation One. Outline. 1: Title 2: Outline 3: Members 4: Mentor 5-6: Societal Issue 7: History 8-9: Dr. Li 10-11: Cluster Computing 12-14: Case Study 15: Accuracy - PowerPoint PPT Presentation
Citation preview
A.R.M.S. Active Resource
Management Services
For Big Data Processing
Revised Presentation One
3/21/2013 1
2
Outline• 1: Title• 2: Outline• 3: Members• 4: Mentor• 5-6: Societal Issue• 7: History• 8-9: Dr. Li• 10-11: Cluster Computing• 12-14: Case Study• 15: Accuracy• 16: Current Major Functional
Component Diagram• 17: Current Process Flow• 18: Problem Statement
• 19: Proposed Major Functional Component Diagram
• 20: Proposed Process Flow• 21-24: Dinosolve Walkthrough• 25: Dinosolve Issues• 26: Software• 27: Hardware• 28: Solution Statement• 29: Competition Identified• 30-32: 508 Compliance• 33: Objectives• 34: Benefits of Solution• 35: Conclusion• 36-39: References• 40-44: Appendix
3/21/2013
Group Members and Roles
• Scott Pardue (Team Leader)• Michael Rajs (Risk Manager)• Adam Willis (Algorithm Specialist)• Sybil Acotanza (Documentation
Specialist)• Jordan Heinrichs (Database Designer)• David Crook (User Interface
Designer)
3/21/2013 3
Dr. Yaohang Li
•Associate Professor in the Department of Computer Science at Old Dominion University.•Research interests include:
•Computational Biology: applies computational simulation techniques to solve biological problems•Markov Chain Monte Carlo (MCMC) methods: statistical algorithm for sampling from probability distributions•Parallel Distributed Grid Computing: uses multiple computers communicating via Internet to solve a problem
3/21/2013 4
How do researchers handle the massive amounts of data they are collecting in
order to benefit their research?
3/21/2013 5
“Every day, [mankind] create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone.”1
3/21/2013 6http://www-01.ibm.com/software/data/bigdata/
7
• Large Hadron Collider 2
– 150 million sensors report 40 million times per second
• Facebook 3
– 2.5 billion – content items shared– 2.7 billion – “Likes”– 300 million – photos uploaded
• Walmart 2
– 1 million customer transactions– 2.5 x 10^15 bytes of data
3/21/2013 http://techcrunch.com/2012/08/22/how-big-is-facebooks-data-2-5-billion-pieces-of-content-and-500-terabytes-ingested-every-day/
Data Management Examples
Dr. Li’s Research• Ideally, his research can be used to
develop new protein-modeling programs. Computational approaches can be more efficient and less expensive than biologists, chemists and others experimenting in lab settings
• Leads to the manufacturing of additional drugs to fight conditions as varied as Alzheimer’s disease, cystic fibrosis and mad cow disease
http://diverseeducation.com/article/13348/
Dr. Li’s Grants
• Dinosolve, his current project, was secured for a five year, $400,000 CAREER Award from the National Science Foundation
• Dr. Li has been the principal or co-principal investigator on research grants totaling more than $15.3 million
Big Data Analysis Hardware• Cluster Computing 4
• A cluster consists of many nodes (computers).• Big data can be generated and analyzed quicker by
spreading the workload amongst the nodes.
3/21/2013 10
Head Node• Logging data• Job submission3 Computation Node• 2 Processors each
• 4 Execution slots per processor
24 total execution slots
Head node packages data from the computation nodes and presents it in a readable format so that it is usable by the research community
Managing the Cluster
Distributed Resource Management Systems (D-RMS)
–Job management subsystem–Physical resource management subsystem–Scheduling and queuing subsystem
3/21/2013 11
12
Dr. Yaohang Li and Dinosolve
• Dinosolve examines a protein sequence of amino acids and determines if the protein can be manipulated by an addition of a disulfide bond
• Each computational result enhances the prediction accuracies for future results
3/21/2013 http://hpcr.cs.odu.edu/dinosolve/index.php
Dinosolve Case Study• Bioinformatics7– Disulfide bond
prediction program
– Disulfide bond creation is important to the research community
3/21/2013 13
Dinosolve Users• Drug design• Pharmaceutical companies
• Antibody design• To combat viruses
• Bio-energy development• Creation of new fuels to replace diminishing
fossil fuels• Genetic mapping5• Research to cure cancer, HIV, and other diseases
3/21/2013 14
Accuracy of Popular Tools
Dinosolve DiANNA Scrath Protein Predictor
Accuracy 90.8% 81% 87%
3/21/2013 Reference 13,14 and 15 15
More users use Dinosolve because of the enhanced accuracy
3/21/2013 17
What is the problem?
• Processing time on big data sets is computationally expensive and as the volume of queries grows the system will progressively drop in performance until the system fails.
• 300 simultaneous requests will cause the web served to crash
3/21/2013 18
3/21/2013 19
3/21/2013 20
3/21/2013 21
User interface will be improved to be more aesthetically pleasing
Working with DinosolveInput titleInput protein sequenceInput e-mail addressSubmit, then wait for confirmation...
Protein Sequence: string of alphabetic characters, each of which represent a particular amino acid in the protein
3/21/2013 22
23
Working with DinosolveConfirmation of requestNow wait for results
3/21/2013
24
Working with DinosolveCheck your e-mail,Click the link providedThe results are displayed
Dinosolve IssuesAs it continues to grow in popularity, these are expected to occur:
• Hard resources for computation– CPU cycles– Memory– Disk space– Network bandwidth
• Server crashes
Goal is to prepare the system to be able to continue to support the research community in light of its expected growth in requests
3/21/2013 25
Software
• Unix operating system installed on the Dinosolve cluster
• Dinosolve algorithm• Sun Grid Engine which will be our
Distributed Resource Management System (D-RMS) installed on the cluster.
• MySQL (database software) • Web-based user interface (website)
3/21/2013 26
Hardware
• MySQL database server• A computer cluster to run the
Dinosolve algorithm• Web server for web-based user
interface
3/21/2013 27
How will we correct the problem?
Configure a distributed resource management system
3/21/2013 28
Competing Distributed Resource Management Systems
• Sun Grid Engine (SGE)• Portable Batch System (PBS)• Load Sharing Facility (LSF)
3/21/2013 29
Dinosolve DiANNA Scrath Protein Predictor
508.22 compliance percentage
67% 85% 67%
3/21/2013 30
508 compliance
• Amended Rehabilitation Act of 1998– require Federal agencies to make their
electronic and information technology accessible to people with disabilities [32]
– enacted to eliminate barriers in information technology, to make available new opportunities for people with disabilities, and to encourage development of technologies that will help achieve these goals [32]
3/21/2013 31
Why is it important to be compliant?
If an entity wishes to receive government funding then any electronic form the entity uses
must be 508 compliant.
3/21/2013 32
Objectives
• Interpret and visualize current usage statistics
• Configure, utilize, and optimize the SGE
• Aesthetically pleasing and professional user interface
3/21/2013 33
What benefits will come from attaining the goals?
• Efficient utilization of available resources• Increased throughput of the cluster• An intuitive and professional user
interface• Rise in popularity due to excellent
accuracy, efficiency, and professional design
3/21/2013 34
Conclusion
With the updated user interface and correctly configured Sun Grid Engine, Dr. Li hopes to establish a
reputable, reliable, and aesthetically pleasing Disulfide
Bonding Prediction Server.
3/21/2013 35
References for history1. http://www-01.ibm.com/software/data/bigdata/2. http://en.wikipedia.org/wiki/Big_data3. http
://techcrunch.com/2012/08/22/how-big-is-facebooks-data-2-5-billion-pieces-of-content-and-500-terabytes-ingested-every-day/
4. http://en.wikipedia.org/wiki/Computer_cluster
3/21/2013 36
References for case study
5. Li, Y. (2010, September 1). CAREER: Novel Sampling Approaches for Protein Modeling Applications [Abstract]. National Science Foundation Award Abstract #1066471.
6. Li, Y., & Yaseen, A. (2012). Enhancing Protein Disulfide Bonding Prediction Accuracy with Context-based Features. Biotechnology and Bioinformatics Symposium
7. bioinformatics. 2011. In Merriam-Webster.com. Retrieved February 15, 2013, from http://www.merriam-webster.com/dictionary/bioinformatics
8. Cronk, J. D. (2012). Disulfide Bond. Retrieved February 15, 2013, from Biochemistry Dictionary: http://guweb2.gonzaga.edu/faculty/cronk/biochem/D-index.cfm?definition=disulfide_bond
9. Yan, Y., & Chapman, B. (2008). Comparative Study of Distributed Resource Management Systems–SGE, LSF, PBS Pro, and LoadLeveler. Technical Report-Citeseerx.
10. Li, Y., & Yaseen, A. (2012). Dinosolve. Retrieved from http://hpcr.cs.odu.edu/dinosolve/
3/21/2013 37
References for competition11. Arvind Krishna, “Why Big Data? Why Now?”, IBM , 2011 URL: http://almaden.ibm.com/colloquium/resources/Why%20Big%20Data%20Krishna.PDF12. Yonghong Yan, Barbara M. Chapman, Comparative Study of Distributed Resource Management Systems - SGE, LSF, PBS Pro, and LoadLeveler, Department of Computer Science, University of Houston, May 2005 (pdf)13. Dr. Li’s site http://hpcr.cs.odu.edu/dinosolve/14. Scratch Predictor http://scratch.proteomics.ics.uci.edu/15. DiANNA server http://clavius.bc.edu/~clotelab/DiANNA/Portable Batch System (PBS)16. http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12-2.pdf17. http://www.pbsworks.com/SupportDocuments.aspx?AspxAutoDetectCookieSupport=118. http://resources.altair.com/pbs/documentation/support/PBSProRefGuide12-2.pdf19. http://resources.altair.com/pbs/documentation/support/PBSProAdminGuide12-2.pdf20.http://www.pbsworks.com/(S(tykrsyqbemmlf3o5zwrmjrgf))/images/solutions-en-US/PBS-Pro_Datasheet-USA_WEB.pdf21.http://agendafisica.files.wordpress.com/2011/05/pbs.pdfMoab HPC Suite22.http://www.adaptivecomputing.com/publication/420/wppa_open/IBM Platform LSF23.http://public.dhe.ibm.com/common/ssi/ecm/en/dcd12354usen/DCD12354USEN.PDFApache Hadoop with Zookeeper24. http://zookeeper.apache.org/doc/current/zookeeperOver.html25. http://www.cloud-net.org/~swsellis/tech/solaris/performance/doc/blueprints/0102/jobsys.pdf
3/21/2013 38
Reference for 508 Compliance
26. http://en.wikipedia.org/wiki/Section_508_Amendment_to_the_Rehabilitation_Act_of_1973
3/21/2013 39
Appendix• 40: Competition Matrix for Resource Management
Systems• 41-43: 508.22 Compliance Statistics for Dinosolve
3/21/2013 40
Competing Resource Management Systems
Features of systems
PBS LSF SGE
Supported platforms
Unix Unix & NT Unix
Multi-clustersupport
Yes Yes No
System level checkpoint
restart
No Yes Yes
User level checkpoint
restart
No No Yes
Large computational grid support
No No No
Massive Scalability
Yes Yes Yes
Parallel job support with Sun HPC ClusterTools
Loose Integration
Tight Integration Loose Integration
Distribution format of end
product
Source Binary only Binary and Source
Free? Yes No YesPosix 1002.2d
complianceYes No Yes3/21/2013 Reference 19 41
3/21/2013 42
3/21/2013 43
3/21/2013 44