Build Your Cluster with Rocks · Build Your Cluster with Rocks Yu Fu 2011 OSG Summer Workshop, Lubbock, TX Rocks Overview • Based ... • Managed with MySQL and XML

Build Your Cluster with RocksBuild Your Cluster with Rocks

Yu FuYu FuUniversity of FloridaUniversity of Florida

2011 OSG Summer Workshop2011 OSG Summer WorkshopLubbock, TXLubbock, TX

Build Your Cluster with Rocks Yu FuBuild Your Cluster with Rocks Yu Fu 2011 OSG Summer Workshop, Lubbock, TX2011 OSG Summer Workshop, Lubbock, TX

You need a cluster You need a cluster ……


and an efficient tool to build itand an efficient tool to build it……


•• Clusters are cost/performance oriented Clusters are cost/performance oriented computational engines, but are hard to computational engines, but are hard to manage.manage.

•• Cluster management gets linearly harder Cluster management gets linearly harder as it scales out and as more and more as it scales out and as more and more frequent updates come for modern OS.frequent updates come for modern OS.

•• Heterogeneous nodes are a bummer.Heterogeneous nodes are a bummer.



What is Rocks?What is Rocks?•• Rocks Cluster Distribution is a Rocks Cluster Distribution is a Linux Linux

distribution distribution intended for intended for highhigh--performance performance computing clusters. computing clusters.

•• Started by National Partnership for Advanced Started by National Partnership for Advanced Computational Infrastructure and the SCSD Computational Infrastructure and the SCSD in 2000.in 2000.

•• Used by 1800+ registered clusters so far and Used by 1800+ registered clusters so far and many more unregistered clusters.many more unregistered clusters.

•• The biggest registered academic Rocks The biggest registered academic Rocks cluster as of now: 8632 CPUs.cluster as of now: 8632 CPUs.



Rocks OverviewRocks Overview•• Based on Based on CentOSCentOS Linux.Linux.•• Uses standard RPM as package tool.Uses standard RPM as package tool.•• Uses automatically generated custom Uses automatically generated custom RedHatRedHat

kickstartkickstart to control the whole installation.to control the whole installation.•• Is Is NOTNOT ““system imagersystem imager”” based.based.•• All nodes are All nodes are ““installed, not installed, not ““imagedimaged””..•• Managed with Managed with MySQLMySQL and XML.and XML.•• Highly customizable.Highly customizable.•• Supports heterogeneous nodes, even crossSupports heterogeneous nodes, even cross--

install a hybrid of i386 and x86_64. install a hybrid of i386 and x86_64. •• Excellent scalability with Excellent scalability with ““AvalancheAvalanche”” install install

mode based on mode based on BitTorrentBitTorrent p2p transfer tool.p2p transfer tool.


Topology of RocksTopology of Rocks


What You Need To Get StartedWhat You Need To Get Started•• A bunch of x86/x86_64 machines A bunch of x86/x86_64 machines –– you can you can

even build a Rocks cluster on virtual machines.even build a Rocks cluster on virtual machines.•• 1GB memory on each machine for Rocks 5.4.1GB memory on each machine for Rocks 5.4.•• 30GB hard drive for Rocks 5.4 default install.30GB hard drive for Rocks 5.4 default install.•• Two Ethernet ports on one of them Two Ethernet ports on one of them –– the the

frontend / frontend / headnodeheadnode..•• PXE booting feature on worker nodes PXE booting feature on worker nodes –– can use can use

CD or floppy emulators. CD or floppy emulators. •• A network switch to connect them up.A network switch to connect them up.•• Download Rocks ISO images and burn to Download Rocks ISO images and burn to

CDs/DVDs CDs/DVDs –– a single a single ““JumboJumbo”” DVD would meet DVD would meet needs in most startup cases, you can add more needs in most startup cases, you can add more rolls later.rolls later.


Build a Rocks ClusterBuild a Rocks Cluster•• Rocks provides for installation of a cluster Rocks provides for installation of a cluster

with minimum user interaction.with minimum user interaction.•• Steps involvedSteps involved

–– Boot from CD or DVD on the frontend.Boot from CD or DVD on the frontend.–– Answer a few questions.Answer a few questions.–– Get favorite beverage Get favorite beverage ……–– YouYou’’re on your way to a full fledged cluster in re on your way to a full fledged cluster in

2 hours or less: ~1 hour for the frontend 2 hours or less: ~1 hour for the frontend server install, ~30 min for worker node, ~20 server install, ~30 min for worker node, ~20 extra sec for each additional worker node.extra sec for each additional worker node.


LetLet’’s Start!s Start!


Minimum Rolls for a frontend: base, kernel, Minimum Rolls for a frontend: base, kernel, osos, web, web--serverserver









Default Rocks 5.4 frontend server disk partitions:Default Rocks 5.4 frontend server disk partitions:

remainder of the primary diskremainder of the primary disk/export (symbolic link to /state/partition1)/export (symbolic link to /state/partition1)

1 GB1 GBswapswap

4 GB4 GB//varvar

16 GB16 GB//


Manual partitioning:Manual partitioning:Minimum of 16GB for / partition.Minimum of 16GB for / partition.Must have a /export partition.Must have a /export partition.Software Software RAIDsRAIDs are supported, but LVM is not supported as in 5.4.are supported, but LVM is not supported as in 5.4.






Install Worker NodesInstall Worker Nodes•• Run Run ““insertinsert--ethersethers”” on frontend:on frontend:


•• Boot worker node from PXE.Boot worker node from PXE.•• insertinsert--ether will automatically detect it, and start install on it.ether will automatically detect it, and start install on it.•• Worker nodes will be named as computeWorker nodes will be named as compute--<rack #><rack #>--<slot #> <slot #>

by default.by default.






•• FromFrom this point, this point, ““rocksrocks--consoleconsole”” can display can display nodenode’’ss screen.screen.•• Based on VNC, no KVM switch is needed! Based on VNC, no KVM switch is needed!


•• Once a worker nodes is recognized by insertOnce a worker nodes is recognized by insert--ethers, ethers, you can go to the next node.you can go to the next node.

•• Repeat the above procedure, until a whole rack is Repeat the above procedure, until a whole rack is finished.finished.

•• Then do the next rack:Then do the next rack:–– Stop insertStop insert--ethers by F8.ethers by F8.–– ReRe--launch insertlaunch insert--ether with the rack number:ether with the rack number:insertinsert--ethers ethers ----rack 1rack 1

–– Boot up worker nodes in this rack one by one from PXE Boot up worker nodes in this rack one by one from PXE until the last node.until the last node.

•• Repeat until the last rack.Repeat until the last rack.•• Your cluster is built!Your cluster is built!


Customize Your Rocks ClusterCustomize Your Rocks Cluster•• Rocks Rocks kickstartkickstart is generated onis generated on--thethe--fly, controlled by fly, controlled by

the graph XML files:the graph XML files:/export/rocks/install/site/export/rocks/install/site--profiles/5.4/nodesprofiles/5.4/nodes

•• Customization can be done in userCustomization can be done in user--defined defined XMLsXMLs..–– PartitionsPartitions–– PackagesPackages–– Extra filesExtra files–– PrePre--install scriptsinstall scripts–– PostPost--install scriptsinstall scripts

•• Check Check kickstartkickstart before install:before install:/export/rocks/install//export/rocks/install/sbin/kickstart.cgisbin/kickstart.cgi ––c c node_namenode_name




An Example of Custom XML:An Example of Custom XML:<?xml version="1.0" standalone="no"?><?xml version="1.0" standalone="no"?><<kickstartkickstart>><description> Enable SSH </description><description> Enable SSH </description><main> <!<main>  </main>> </main><pre> <!<pre>  </pre>> </pre><package> <package> sshssh </package></package><package> <package> sshssh--clients </package>clients </package><package> <package> sshssh--server </package>server </package><package> <package> sshssh--askpassaskpass </package></package>

<post><post>cat &cat &gtgt; /etc/; /etc/ssh/ssh_configssh/ssh_config &<&ltlt;< 'EOF'; 'EOF'Host *Host *

CheckHostIPCheckHostIP nonoForwardX11 yesForwardX11 yesForwardAgentForwardAgent yesyesStrictHostKeyCheckingStrictHostKeyChecking nonoUsePrivilegedPortUsePrivilegedPort nonoFallBackToRshFallBackToRsh nonoProtocol 1,2Protocol 1,2

EOF </post> EOF </post> </</kickstartkickstart>>


Customize PartitionsCustomize Partitions•• A single drive example:A single drive example:

<pre><pre>echo echo ““clearpartclearpart ----all all ----initlabelinitlabel ----drives=drives=sdasdapart / part / ----size 8000 size 8000 ----ondiskondisk sdasdapart swap part swap ----size 1000 size 1000 ----ondiskondisk sdasdapart /data part /data ----size 0 size 0 ----grow grow ----ondiskondisk sdasda”” &&gtgt; ;

//tmp/user_partition_infotmp/user_partition_info</pre></pre>

•• Software RAIDSoftware RAID<pre><pre>echo echo ““clearpartclearpart ----all all ----initlabelinitlabel ----drives=drives=sda,sdbsda,sdbpart / part / ----size 8000 size 8000 ----ondiskondisk sdasdapart swap part swap ----size 1000 size 1000 ----ondiskondisk sdasdapart raid.00 part raid.00 ----size=20000 size=20000 ----ondiskondisk sdasdapart raid.01 part raid.01 ----size=20000 size=20000 ----ondiskondisk sdbsdbraid /data raid /data ----level=1 level=1 ----device=md0 raid.00 raid.01device=md0 raid.00 raid.01”” &&gtgt; ;

/tmp/user_partition_info/tmp/user_partition_info</pre></pre>

•• Manual partitioningManual partitioningecho echo ““rocks manualrocks manual”” &&gtgt; /; /tmp/user_partition_infotmp/user_partition_info


Install Extra Software PackagesInstall Extra Software Packages

•• All Rocks packages are installed and managed All Rocks packages are installed and managed through RPM.through RPM.–– Prepare the RPMPrepare the RPM–– Put it in Put it in /export/rocks/install/contrib/5.4//export/rocks/install/contrib/5.4/archarch/RPMS/RPMS

–– Add it in the XML:Add it in the XML:<package> <package> your_packageyour_package </package></package>

–– ReRe--create the Rocks create the Rocks distrodistro::# # cdcd /export/rocks/install /export/rocks/install # rocks create # rocks create distrodistro


Update a SoftwareUpdate a Software•• The same method can also be used to update a The same method can also be used to update a

software in the Rocks. Rocks will automatically software in the Rocks. Rocks will automatically pickup and install the newer version on worker pickup and install the newer version on worker nodes.nodes.

•• Get the RPM of the new version, e.g.:Get the RPM of the new version, e.g.:wgetwget ftp://ftp://ftp.scientificlinux.org/linux/scientificftp.scientificlinux.org/linux/scientific/5x/x86_64/SL/kernel/5x/x86_64/SL/kernel--2.6.182.6.18--238.9.1.el5.x86_64.rpm238.9.1.el5.x86_64.rpm

•• Put it in Put it in /export/rocks/install/contrib/5.4//export/rocks/install/contrib/5.4/archarch/RPMS/RPMS

or or /export/rocks/install/rolls/os/5.4//export/rocks/install/rolls/os/5.4/archarch/RedHat/RPMS/RedHat/RPMS

•• ReRe--create the Rocks create the Rocks distrodistro::# # cdcd /export/rocks/install /export/rocks/install # rocks create # rocks create distrodistro


NonNon--RPM SoftwareRPM SoftwareTo share files/software not available in RPM, To share files/software not available in RPM, you can put them in Rocks system NFS:you can put them in Rocks system NFS:–– On the On the frontendfrontend, go to the directory , go to the directory

/share/apps./share/apps.–– Add the files you'd like to share within this Add the files you'd like to share within this

directory.directory.–– All files there will be available on the compute All files there will be available on the compute

nodes under: /share/appsnodes under: /share/apps


Custom Custom ConfigConfig ScriptsScripts•• PostPost--install custom install custom configconfig scripts can be put in the <post> scripts can be put in the <post>

</post> section of the </post> section of the XMLsXMLs..•• Just like regular Shell scripts, but be careful with XML Just like regular Shell scripts, but be careful with XML

reserved characters, e.g. replace reserved characters, e.g. replace ““>>”” with with ““&&gtgt;;”” etc.etc.<post><post>cat &cat &gtgt; /etc/; /etc/ssh/ssh_configssh/ssh_config &<&ltlt;< 'EOF'; 'EOF'Host *Host *

CheckHostIPCheckHostIP nonoForwardX11 yesForwardX11 yesForwardAgentForwardAgent yesyesStrictHostKeyCheckingStrictHostKeyChecking nonoUsePrivilegedPortUsePrivilegedPort nonoFallBackToRshFallBackToRsh nonoProtocol 1,2Protocol 1,2

EOF EOF </post></post>

•• Other scripts such as Python are also possible.Other scripts such as Python are also possible.


Rocks DatabaseRocks Database

MySQL DB

makehosts

/etc/hosts

makedhcp

/etc/dhcpd.conf

insert-ethers

Node 0

Node 1

Node N

Automated nodediscovery

Kickstart.cgiXMLs

kickstart


Older versions of Rocks had a convenient webOlder versions of Rocks had a convenient web--based tool to manage its based tool to manage its MySQLMySQLdatabase. Rocks 5 provides handy commands to deal with databasedatabase. Rocks 5 provides handy commands to deal with database--related related tasks and no longer encourages user to directly access its tasks and no longer encourages user to directly access its MySQLMySQL. .


Rocks Rocks ‘‘n Rollsn Rolls•• Rolls are Rocks Modules.Rolls are Rocks Modules.•• It is the mechanism for delivery of packages and It is the mechanism for delivery of packages and

configuration in Rocks.configuration in Rocks.•• What is inside a roll?What is inside a roll?

–– Packaged binaries: Packaged binaries: RPMsRPMs–– Automatic configuration dataAutomatic configuration data–– Installation mapInstallation map–– A roll is basically a collection of A roll is basically a collection of RPMsRPMs plus Kickstart graph plus Kickstart graph XMLsXMLs

packed in the ISO form.packed in the ISO form.

•• There are tons of rolls readily available: ganglia, There are tons of rolls readily available: ganglia, hpchpc, bio, , bio, sgesge, torque, condor, , torque, condor, xenxen …………

•• You can build your own roll!You can build your own roll!•• As OSG releases are moving to RPM, we may have an OSG As OSG releases are moving to RPM, we may have an OSG

roll in the future!roll in the future!


Rocks 411 ServiceRocks 411 Service

•• The 411 Secure The 411 Secure Information Service Information Service provides NISprovides NIS--like like functionality for Rocks functionality for Rocks clusters with better clusters with better security and scalability.security and scalability.

•• It is basically a file sync It is basically a file sync mechanism with listener mechanism with listener and poll agents. and poll agents.

•• Unlike NIS, changes must Unlike NIS, changes must be made on be made on frontendfrontend..


CrossCross--architecture Installarchitecture Install

•• Rocks is able to Rocks is able to heterogeneousheterogeneous nodes nodes even with different architectures.even with different architectures.

•• Rocks can install i386 nodes from an Rocks can install i386 nodes from an x86_64 x86_64 frontendfrontend..

•• Rocks can install x86_64 nodes from an Rocks can install x86_64 nodes from an i386 i386 frontendfrontend..

•• It is quite simple to do so in Rocks 5.4, It is quite simple to do so in Rocks 5.4, see details in Usersee details in User’’s Guide s Guide http://www.rocksclusters.org/rollhttp://www.rocksclusters.org/roll--documentation/base/5.4/cross.htmldocumentation/base/5.4/cross.html


Rocks Upgrade Rocks Upgrade –– Keep Keep ConfigsConfigs•• Very simple!Very simple!

1.1. Create a backup roll of current Create a backup roll of current custom version:custom version:

# # cdcd /export/site/export/site--roll/rocks/roll/rocks/srcsrc/roll/restore /roll/restore

# make roll# make roll

2.2. Install a clean new version.Install a clean new version.3.3. Include the backup roll at the Include the backup roll at the

beginning of the install.beginning of the install.4.4. Done!Done!

•• It is also a handy way to It is also a handy way to back up the back up the frontendfrontend..


Further InformationFurther Information

•• Rocks UserRocks User’’s Guide: s Guide: http://www.rocksclusters.org/rollhttp://www.rocksclusters.org/roll--documentation/base/5.4/documentation/base/5.4/

•• Rocks Rocks WikiWiki: : https://wiki.rocksclusters.orghttps://wiki.rocksclusters.org•• Rocks UserRocks User’’s Mailing List: s Mailing List:

https://https://lists.sdsc.edu/mailman/listinfo/npalists.sdsc.edu/mailman/listinfo/npacici--rocksrocks--discussiondiscussion


DemoDemo


Questions?Questions?

Documents

Build Your Cluster with Rocks · Build Your Cluster with Rocks Yu Fu 2011 OSG Summer Workshop, Lubbock, TX Rocks Overview • Based ... • Managed with MySQL and XML