86
Linux-HA Japan Project 1 痛い目にあってわかる HAクラスタのありがたさ Pacemakerで心に安らぎを~ 20101127OSC2010 Shimane Linux-HA Japan 松尾隆利

痛い目にあってわかる HAクラスタのありがたさ

Embed Size (px)

DESCRIPTION

OSC 2010 島根 Flash -> http://linux-ha.sourceforge.jp/wp/archives/693

Citation preview

  • 1. HA Pacemaker 20101127 OSC2010 ShimaneLinux-HA Japan Linux-HA Japan Project1

2. HAPacemaker Pacemaker Pacemaker Pacemaker Linux-HA Japan Linux-HA Japan Project2 3. HA PacemakerLinux-HA Japan Project3 4. (Wikipedia) (Wikipedia)HA HPC Linux-HA Japan Project4 5. HA High Availability = Linux-HA Japan Project5 6. HA Heartbeat Heartbeat HA Linux-HA Japan Project6 7. Pacemaker Linux-HA Japan Project7 8. HA Heartbeat Linux-HA Japan Project8 9. Pacemaker Heartbeat Linux-HA Japan Project9 10. Linux-HA Japan Project10 11. Active/Standby1+1 Active ActiveStandbyActive LANActiveActiveStandby Linux-HA Japan Project 11 12. Pacemaker Heartbeat1Active Standby N+M Active2 2+1 Active1Active2Active1StandbyActive2 Active2 Linux-HA Japan Project12 13. Active / Standby 1+1 Linux-HA Japan Project13 14. PacemakerLinux-HA Japan Project14 15. Pacemaker 1. 2. Linux-HA Japan Project15 16. Active PacemakerStandbyActiveStandby PacemakerLAN LAN LAN LAN Linux-HA Japan ProjectPacemakerPacemaker 16 17. (RA) (start)(stop)(monitor) ActiveStandbyActiveRARARAPacemakerPacemakerPacemakerOK OK RA Pacemaker Linux-HA Japan ProjectStandbyStandby Standby Standby 17 18. Pacemaker Pacemaker Linux-HA Japan Project18 19. IP IP , IP : 192.168.0.10ActivemountStandbyNFSLinux-HA Japan Project19 20. (RA) Pacemaker ActivePacemaker (start) (stop) (monitor) Linux-HA Japan ProjectPacemaker PostgreSQL PostgreSQL RA PostgreSQL 20 21. (Pacemaker PostgreSQL Pacemaker PacemakerPostgreSQL PostgreSQL (pgsql) (pgsql)start OK/NG pgsql_start()monitor OK/NG pgsql_monitor()stop OK/NG pgsql_stop() Linux-HA Japan ProjectPostgreSQL PostgreSQL PostgreSQLpg_ctl start OK/NGselect now(); OK/NGpg_ctl stop OK/NG 21 22. Pacemaker Heartbeat Heartbeat Heartbeat Heartbeat RA RA RA RA RA RA PostgreSQL Apache, Tomcat IP IP (/usr/lib/ocf/resource.d/ )pgsql apache, tomcat, Filesystem () IPaddr, IPaddr2 diskd (Linux-HA Japan) pingd Linux-HA Japan Project22 23. Pacemaker Linux-HA Japan Project23 24. Pacemaker Linux-HA Japan Project24 25. : Pacemaker Heartbeat Heartbeat Tengine CRM Pengine CCM CRM: Tengine: Pengine: CCM: RA:RA Heartbeat PacemakerPacemakerCluster Resource Manager Transition Engine Policy engine Cluster Consensus Membership Linux-HA Japan Project Resource Agent25 26. HA Heartbeat Tengine CRM Pengine CCMRA Linux-HA Japan Project26 27. 1 : Heartbeat Heartbeat Tengine CRM Pengine CCMHeartbeat Heartbeat RAHeartbeat Linux-HA Japan Project27 28. 2 : Corosync OpenAIS OpenAIS HA CorosyncCorosync Linux-HA Japan Project28 29. Pacemaker Heartbeat3Corosync Linux-HA Japan Project29 30. Linux-HA Japan HA Heartbeat HeartbeatPacemaker Heartbeat HeartbeatPacemaker Pacemaker + Heartbeat3 Pacemaker + Corosync OpenAIS + CorosyncOpenAISOpenAIS Corosync Linux-HA Japan Project 30 31. Heartbeat3 Corosync Linux-HA Japan Project31 32. Heartbeat Heartbeat2 7LAN() (3) Linux-HA Japan Project32 33. Corosync 11 16 Linux-HA Japan Project33 34. Pacemaker CentOS 5.5 x86_64 Linux-HA Japan Project34 35. 1.yum Pacemaker(clusterlabs) yum 2. + yum Linux-HA Japan Linux-HA Japan 3.rpm rpm4. Linux-HA Japan Project35 36. Pacemaker rpm 20101119HA RHEL5, CentOS5 x86_64rpm 9 pacemaker-1.0.10-1.4.el5.x86_64.rpm pacemaker-libs-1.0.10-1.4.el5.x86_64.rpm corosync-1.2.7-1.1.el5.x86_64.rpm corosynclib-1.2.7-1.1.el5.x86_64.rpm cluster-glue-1.0.6-1.el5.x86_64.rpm cluster-glue-libs-1.0.6-1.el5.x86_64.rpm resource-agents-1.0.3-2.6.el5.x86_64.rpm heartbeat-3.0.3-2.3.el5.x86_64.rpm heartbeat-libs-3.0.3-2.3.el5.x86_64.rpm Linux-HA Japan Project11/12 1.0.10 11/12 1.0.10 CorosyncHeartbeat3 CorosyncHeartbeat3 CorosyncHeartbeat3 CorosyncHeartbeat3 rpm rpm rpm rpm rpm rpm 36 37. yum 1 . epel yum CentOSrpm Fedorayum # wget http://download.fedora.redhat.com/pub/epel/5/x86_64/epel# wget http://download.fedora.redhat.com/pub/epel/5/x86_64/epelrelease-5-4.noarch.rpm release-5-4.noarch.rpm # rpm -ivh epel-release-5-4.noarch.rpm # rpm -ivh epel-release-5-4.noarch.rpmLinux-HA Japan Project37 38. yum 2. Pacemakeryum clusterlabs.org yumrepo # cd /etc/yum.repo.d yum # cd /etc/yum.repo.d yum # wget http://clusterlabs.org/rpm/epel-5/clusterlabs.repo # wget http://clusterlabs.org/rpm/epel-5/clusterlabs.repoLinux-HA Japan Project38 39. yum 3. # yum install pacemaker.x86_64 # yum install pacemaker.x86_64 rpm pacemaker-libs (clusterlabs) corosync (clusterlabs) corosynclib (clusterlabs) cluster-glue (clusterlabs) cluster-glue-libs (clusterlabs) resource-agents (clusterlabs) heartbeat (clusterlabs) heartbeat-libs (clusterlabs) libesmtp (epel) Linux-HA Japan Project39 40. + yum OK1. Pacemaker Linux-HA Japan Pacemaker pacemaker-1.0.10-1.4.1.el5.x86_64.repo.tar.gzPacemaker-1.0.10 Pacemaker-1.0.10 (11/26) (11/26) Linux-HA Japan Project40 41. + yum 2. Pacemaker # cd /tmp /tmp # cd /tmp /tmp # tar zxvf ///pacemaker-1.0.10-1.4.1.el5.x86_64.repo.tar.gz # tar zxvf ///pacemaker-1.0.10-1.4.1.el5.x86_64.repo.tar.gz3. yum # cd pacemaker-1.0.10-1.4.1.el5.x86_64.repo # cd pacemaker-1.0.10-1.4.1.el5.x86_64.repo # yum c pacemaker.repo install pacemaker # yum c pacemaker.repo install pacemakerLinux-HA Japan yum pm_crmgen-1.0-1.el5.noarch.rpm crm pm_diskd-1.0-1.el5.x86_64.rpm RA pm_logconv-hb-1.0-1.el5.noarch.rpm pm_extras-1.0-1.el5.x86_64.rpm Linux-HA Japan RA Project41 42. RHEL 5.5 CentOS 5.5 libxml2 Pacemaker libxml2-2.6.26-2.1.2.8.el5_5.1 : https://rhn.redhat.com/errata/RHBA-2010-0764.htmlLinux-HA Japan Project42 43. Pacemaker Pacemaker + Heartbeat3 Linux-HA Japan Project43 44. Pacemaker [ Pacemaker ] [ Heartbeat3 ] Linux-HA Japan Project44 45. /etc/ha.d/ha.cf pacemaker on pacemaker on debug 0 debug 0 udpport 694 udpport 694 keepalive 2 keepalive 2 warntime 20 warntime 20 deadtime 24 deadtime 24 initdead 48 initdead 48 logfacility local1 logfacility local1 Heartbeat Heartbeat ha.cf ha.cf crm onpacemaker on bcast eth1 bcast eth1 bcast eth3 bcast eth3eth1node pm01 node pm01 node pm02 node pm02 watchdog /dev/watchdog watchdog /dev/watchdogeth3pm01Linux-HA Japan Projectpm0245 46. /etc/ha.d/authkeys / root/root rw---- Heartbeat Heartbeat auth 1 auth 1 1 sha1 oscshimane 1 sha1 oscshimane sha1, md5, crcLinux-HA Japan Project46 47. /etc/syslog.conf /var/log/messages /var/log/ha-log *.info;mail.none;authpriv.none;cron.none;local1.none *.info;mail.none;authpriv.none;cron.none;local1.none /var/log/messages /var/log/messages local1.* /var/log/ha-log local1.* /var/log/ha-log ha.cf logfacility Linux-HA Japan Project47 48. Pacemaker # /etc/init.d/heartbeat start # /etc/init.d/heartbeat start Starting High-Availability services: Starting High-Availability services:Linux-HA Japan Project [[ OK ]] OK48 49. Pacemaker crm_mon # crm_mon # crm_mon ============ ============ Last updated: Wed Nov 10 14:28:55 2010 Last updated: Wed Nov 10 14:28:55 2010 Stack: Heartbeat Stack: Heartbeat Current DC: pm02 (a59a9306-d6e7-4357-bb0c-a5aea0615e61) --partition Current DC: pm02 (a59a9306-d6e7-4357-bb0c-a5aea0615e61) partition with quorum with quorum Version: 1.0.10-89bd754939df5150de7cd76835f98fe90851b677 Version: 1.0.10-89bd754939df5150de7cd76835f98fe90851b677 2 Nodes configured, unknown expected votes 2 Nodes configured, unknown expected votes 0 Resources configured. 0 Resources configured. ============ ============ Online: [ [pm02 pm01 ] ] Online: pm02 pm01 () () Linux-HA Japan Project Linux-HA Japan Project49 49 50. Linux-HA Japan Project50 51. ApachePostgreSQLNW (RA) monitor (RA) Linux-HA Japan Project51 52. crm Pacemakercib.xml XML Heartbeat 2 xml Linux-HA Japan Project52 53. crm Pacemaker TABcrm resource node options configure status help exitshow : cleanup status : standby show : crm edit load : commit crm crm 53 Linux-HA Japan Project 54. crm IP + Apache apache IP : 172.20.24.110Apache eth0eth0IP : 172.20.24.110Apacheeth1 eth3 : pm01 : pm02 Linux-HA Japan Project54 55. crm : crm # crm # crm crm(live)# configure crm(live)# configure crm(live)configure# crm(live)configure# # crm configure # crm configure crm(live)configure# crm(live)configure# Linux-HA Japan Project55 56. crm : 2ignoreSTONITH (false) STONITH crm(live)configure# property no-quorum-policy="ignore" crm(live)configure# property no-quorum-policy="ignore" stonith-enabled="false" stonith-enabled="false" startup-fencing="false" startup-fencing="false" Linux-HA Japan Project56 57. crm : IP IP IPaddr2 IPID MyIp ()crm(live)configure# primitive MyIp ocf:heartbeat:IPaddr2 crm(live)configure# primitive MyIp ocf:heartbeat:IPaddr2 params params ip="172.20.24.110"" ip="172.20.24.110 IPaddr2 nic="eth0" nic="eth0" cidr_netmask="24 cidr_netmask="24 op start interval="0s" timeout="60s" on-fail="restart" op start interval="0s" timeout="60s" on-fail="restart" op monitor interval="10s" timeout="60s" on-fail="restart" op monitor interval="10s" timeout="60s" on-fail="restart" op stop interval="0s" timeout="60s" on-fail="block" op stop interval="0s" timeout="60s" on-fail="block"Linux-HA Japan Project57 58. PrimitivePrimitivePrimitivePrimitive () PostgreSQL, IP () PostgreSQL, IPCloneCloneCloneCloneMaster / SlaveMasterSlaveSlave Linux-HA Japan Project Primitive Primitive Clone Clone () NW, () NW, Primitive Primitive Maseter/Slave Maseter/Slave RAMasterSlave RAMasterSlave () DRBD 58 () DRBD 59. crm : Apache apacheApache ApacheID MyApache()crm(live)configure# primitive MyApache ocf:heartbeat:apache crm(live)configure# primitive MyApache ocf:heartbeat:apache params params statusurl="http://localhost/test.html" apache testregex="hogehoge" httpd="/usr/sbin/httpd" httpd="/usr/sbin/httpd" configfile="/etc/httpd/conf/httpd.conf" configfile="/etc/httpd/conf/httpd.conf" op start interval="0s" timeout="60s" on-fail="restart" op start interval="0s" timeout="60s" on-fail="restart" op monitor interval="10s" timeout="60s" on-fail="restart" op monitor interval="10s" timeout="60s" on-fail="restart" op stop interval="0s" timeout="60s" on-fail="block"" op stop interval="0s" timeout="60s" on-fail="block ApacheURL # echo "hogehoge" > /var/www/html/test.html Linux-HA Japan Project # echo "hogehoge" > /var/www/html/test.html59 60. crm : IPaddr2apache IDMyGrp ()crm(live)configure# group MyGrp MyIp MyApache crm(live)configure# group MyGrp MyIp MyApache Linux-HA Japan Project60 61. crm : crm(live)configure# show crm(live)configure# show node $id="a0dacbcf-346f-4003-ab5b-15422e0e4697" pm01 node $id="a0dacbcf-346f-4003-ab5b-15422e0e4697" pm01 node $id="fe705a39-541a-4b10-af22-de27d4c72d23" pm02 node $id="fe705a39-541a-4b10-af22-de27d4c72d23" pm02apacheprimitive MyApache ocf:heartbeat:apache primitive MyApache ocf:heartbeat:apache params statusurl="http://localhost/test.html" testregex="hogehoge" params statusurl="http://localhost/test.html" testregex="hogehoge" httpd="/usr/sbin/httpd" configfile="/etc/httpd/conf/httpd.conf" httpd="/usr/sbin/httpd" configfile="/etc/httpd/conf/httpd.conf" op start interval="0s" timeout="60s" on-fail="restart" op start interval="0s" timeout="60s" on-fail="restart" op monitor interval="10s" timeout="60s" on-fail="restart" op monitor interval="10s" timeout="60s" on-fail="restart" op stop interval="0s" timeout="60s" on-fail="block" op stop interval="0s" timeout="60s" on-fail="block"IPprimitive MyIp ocf:heartbeat:IPaddr2 primitive MyIp ocf:heartbeat:IPaddr2 params ip="172.20.24.110" nic="eth0" cidr_netmask="24" params ip="172.20.24.110" nic="eth0" cidr_netmask="24" op start interval="0s" timeout="60s" on-fail="restart" op start interval="0s" timeout="60s" on-fail="restart" op monitor interval="10s" timeout="60s" on-fail="restart" op monitor interval="10s" timeout="60s" on-fail="restart" op stop interval="0s" timeout="60s" on-fail="block" op stop interval="0s" timeout="60s" on-fail="block"group MyGrp MyIp MyApache group MyGrp MyIp MyApache property $id="cib-bootstrap-options" property $id="cib-bootstrap-options" dc-version="1.0.9-89bd754939df5150de7cd76835f98fe90851b677" dc-version="1.0.9-89bd754939df5150de7cd76835f98fe90851b677" cluster-infrastructure="Heartbeat" cluster-infrastructure="Heartbeat" no-quorum-policy="ignore" no-quorum-policy="ignore" stonith-enabled="false" stonith-enabled="false" startup-fencing="false" Linux-HA Japan Project startup-fencing="false"61 62. crm : commit crm(live)configure# commit crm(live)configure# commit crm(live)configure# save /root/config.crm crm(live)configure# save /root/config.crm Pacemaker /var/lib/heartbeat/crm/ crm(live)configure# load update /root/config.crm crm(live)configure# load update /root/config.crm Linux-HA Japan Project62 63. crm_mon # crm_mon # crm_mon============ ============ ============ ============IPApache pm02Online: [ [pm02 pm01 ] ] Online: pm02 pm01 Resource Group: MyGrp Resource Group: MyGrp MyIp (ocf::heartbeat:IPaddr2): MyIp (ocf::heartbeat:IPaddr2): MyApache (ocf::heartbeat:apache): MyApache (ocf::heartbeat:apache): Linux-HA Japan ProjectStarted pm02 Started pm02 Started pm02 Started pm02 63 64. pm02# poweroff -nf eth0IP : 172.20.24.110 Apache eth1 eth3pm01pm02 Linux-HA Japan Project64 65. ============ ============ ============ ============ Online: [ [pm01 ] ] Online: pm01 OFFLINE: [ [pm02 ] ] OFFLINE: pm02pm01pm02 pm01pm02 OFFLINE OFFLINEResource Group: MyGrp Resource Group: MyGrp MyIp (ocf::heartbeat:IPaddr2): MyIp (ocf::heartbeat:IPaddr2): MyApache MyApache (ocf::heartbeat:apache): (ocf::heartbeat:apache):Started pm01 Started pm01 Started pm01 Started pm01pm01 pm01 Linux-HA Japan Project65 66. Apache pm02# rm /var/www/html/test.htmlIP : 172.20.24.110eth0Apache eth1 eth3pm01pm02 Linux-HA Japan Project66 67. ============ ============ ============ ============pm0 1 pm01 pm0 pm0Online: [ [pm02 pm01 ] ] Online: pm02 pm01 Resource Group: MyGrp Resource Group: MyGrp MyIp (ocf::heartbeat:IPaddr2): MyIp (ocf::heartbeat:IPaddr2): MyApache (ocf::heartbeat:apache): MyApache (ocf::heartbeat:apache):Started pm01 Started pm01 Started pm01 Started pm01 pm02 pm02 pm02 pm02 pm02 pm02 Apache Apache Apache Apache monitor, start monitor, start Failed actions: Failed actions: MyApache_monitor_10000 (node=pm02, call=13, rc=1, status=complete): unknown error MyApache_monitor_10000 (node=pm02, call=13, rc=1, status=complete): unknown error MyApache_start_0 (node=pm02, call=15, rc=1, status=complete): unknown error MyApache_start_0 (node=pm02, call=15, rc=1, status=complete): unknown errorLinux-HA Japan Project67 68. Active NW NW Linux-HA Japan Project68 69. Linux-HA Japan Project69 70. crm Linux-HA Japan Project70 71. pm_crmgen Linux-HA Japan Linux-HA Japan crm crm Excel crm http://sourceforge.jp/projects/linux-ha/ Linux-HA Japan Project71 72. pm_crmgen rpm # rpm ivh pm_crmgen-1.0-1.el5.noarch.rpm # rpm ivh pm_crmgen-1.0-1.el5.noarch.rpm # yum c pacemaker.repo pm_crmgen # yum c pacemaker.repo pm_crmgenLinux-HA Japan Project72 73. Excel /usr/share/pacemaker/pm_crmgen/pm_crmgen_env.xlsExcel PC IPaddr2 IPaddr2 IP IP Linux-HA Japan Project 73 74. Active ActiveStandby ActiveStandby ActiveStandby ActiveStandby ID ID Linux-HA Japan Project74 75. crm CSVIPActivecrm_sample.csv crm_sample.csv CSV CSV CSV CSVSCPFTPpm_crmgen Linux-HA Japan Project75 76. crm pm_crmgencrm # pm_crmgen o crm_sample.crm crm_sample.csv # pm_crmgen o crm_sample.crm crm_sample.csv CSV crm # crm configure load update crm_sample.crm # crm configure load update crm_sample.crmLinux-HA Japan Project76 77. Linux-HA Japan Linux-HA Japan Project77 78. Linux-HA Japan Heartbeat() 2007105Linux-HA (Heartbeat) Linux-HA HeartbeatrpmHeartbeat Linux-HA Japan Project78 79. Linux-HA Japan URL http://linux-ha.sourceforge.jp/ () http://sourceforge.jp/projects/linux-ha/ () 6/25 Linux-HA Japan Project79 80. Linux-HA Japan HA Linux-HA Japan Linux-HA-Japan MLPacemakerHeartbeatCorosync DRBDHA MLURL http://linux-ha.sourceforge.jp/ http://linux-ha.sourceforge.jp/ ML linux-ha- [email protected] [email protected] .j Linux-HA Japan Project80 81. Pacemaker clusterlabs.org http://clusterlabs.org/ Fedora, openSUSE, Fedora, openSUSE, EPEL(CentOS/RHEL) EPEL(CentOS/RHEL) rpm rpm Linux-HA Japan Project81 82. PacemakerLinux-HA Japan Project82 83. Linux-HA Japan Project83 84. Pacemaker Linux-HA Japan PacemakerLinux-HA Japan Project84 85. Pacemaker : http://theclusterguy.clusterlabs.org/post/1551578523/new-logo Cluster Guy New logo VS11/24 11/24 Linux-HA Japan Project85 86. Linux-HA Japanhttp://linux-ha.sourceforge.jp/ Linux-HA Japan Project86