frontier rpms 1

Frontier rpms

June 2011David Front

Weizmann Institute

8 June 2011

frontier rpms 28 June 2011

frontier rpms 3

Why rpms?

Once a site (takes the effort and) moves to use rpms• upgrading SW is faster, and easier compared to using tarballs• It should also be more safe … assuming that proper testing of

the new rpm release has been done

Yum package manager is supported by frontier rpms• With yum, installing, upgrading and removing is easier that via

rpm: - yum is aware of rpm dependencies- No need to specify rpm version, only package name

• Automatic upgrade possibility:- Periodically do ‘yum upgrade package’ (via crontab)

• Note, however, that relocation is not supported by yum8 June 2011

frontier rpms 4

Frontier rpms – history and status• Frontier has been written by CMS and distributed as tarballs• Favia Donno did create initial frontier rpms for ATLAS:

– frontier-squid– frontier-tomcat– frontier-servlet– frontier-awstats

• A new release of frontier rpms is available, is in use at CERN ATLAS product servers since today, and is planned to be defined as ‘production ready’ soon:– In particular, there is a need for rpms to be tested out of CERN first

• Current use of frontier rpms:– ATLAS (and CMS?) has squids installed as rpms– The other frontier rpms are installed only at ATLAS servers at CERN:

CMS, and non CERN ATLAS frontier servers are installed via tarballs.

8 June 2011

frontier rpms 5

Installing frontier rpms: Prerequisites• The (configurable) user to own rpm files

should exist prior to rpm installation: Default: – ‘squid’ for squid – ‘dbfrontier’ for the other rpms

• If you are installing frontier rpms, instead of existing non rpm (tarball) package/s:

First uninstall the existing non rpm SW: – Remove the existing directory in which the software was installed – Take care to remove/undo changes at other files that have been edited

manually or by scripts. In particular, undo crontab (manual) changes.

– Verify that squid/tomcat service actually stopped:Be aware that: Stopping squid/tomcat may take a long time, or:

‘service frontier-{squid or tomcat} status’ may say by mistake ‘stopped’,while ‘ps –efw|grep {squid or tomcat}’ is not empty.

8 June 2011

frontier rpms 6

Installing frontier rpms: configuration

• Manually take care of per rpm and per site and per site/machine files, if any. For details, consult later slides, and:

http://frontier.cern.ch/dist/rpms/_README

• Configuration changes: As a rule of thumb, configuration changes are applied to files as

follows: • The directory is /etc/{squid|awstats(|tomcat)} • Configuration changes of customize.sh are applied when the

appropriate server restarts. • Hence, there should be no need to reinstall an rpm in order to apply

(such) configuration changes. • In addition, for frontier-squid, squidconf is applied at install time

8 June 2011

frontier rpms 7

Installing frontier rpms (cont.)• In order to install frontier rpms, as user root,

import the gpg public key once (per machine): rpm --import http://frontier.cern.ch/dist/rpms/cernFrontierGpgPublicKey

• Using the command 'rpm': as user root: rpm -Uvh [--prefix <RPM_INSTALL_PREFIX>] http://frontier.cern.ch/dist/rpms/<rpm>

• Relocating an rpm: The defaults of RPM_INSTALL_PREFIX are:

frontier-squid: / frontier-awstats: / frontier-tomcat: /data/dbfrontier frontier-servlet: /data/dbfrontier/tomcat/webapps/atlr (for servlet atlr)

The frontier rpms are relocatableException: frontier-tomcat and frontier-servlet rpms can not be relocated to /The plan is that / will be the default for them as well.

8 June 2011

frontier rpms 8

The current rpm versions

At: http://frontier.cern.ch/dist/rpms/:• frontier-awstats-6.0-1.noarch.rpm• frontier-servlet-3.29-1.noarch.rpm • frontier-tomcat-6.0.32-1.noarch.rpm • frontier-squid-2.7.STABLE9-5.3.sl5.x86_64.rpm

8 June 2011

frontier rpms 9

Installing via yum

• get cernFrontier repository file once per machine: wget -O /etc/yum.repos.d/cern-frontier.repo http://frontier.cern.ch/dist/rpms/cern-frontier.repo

• Now you may use yum as usual. for example: yum install package1 [package2] [...] yum upgrade package1 [package2] [...]yum remove package1 [package2] [...]

8 June 2011

frontier rpms 10

Testing the installation: tomcat, squid

For a simple test of tomcat and squid servers, using fnget.py:you may adjust and run commands like the following, from bash:

wget -O /tmp/fnget.py http://frontier.cern.ch/dist/fnget.py chmod +x /tmp/fnget.py server=atlasfrontier1.cern.ch # Adjust this to your serverservlet=atlr # Adjust this to you servlet port=8000 # To test tomcat. Adjust if needed /tmp/fnget.py \

--url='http://'${server}':'${port}'/'${servlet}'/Frontier' \--sql="select 'TOMCAT_IS_HAPPY' from dual" \--refresh-cache

port=8080 # To test squid. Adjust if needed

/tmp/fnget.py \--url='http://'${server}':'${port}'/'${servlet}'/Frontier‘ \--sql="select 'SQUID_IS_HAPPY' from dual“ \--refresh-cache

If all went well, you should get records like the following: 'TOMCAT_IS_HAPPY' CHAR 'SQUID_IS_HAPPY' CHAR

8 June 2011

frontier rpms 11

Testing the installation: awstats, logrotate

To see that awstats works,one may wait a while and see that the numbers or the

upgraded squid do grow

Logrotates:Logs should not grow too much:

/data/tomcat_logs/catalina.out /data/tomcat_logs/modifiedqueries.out/data/squid_logs/access.log

To see the allowed sizes:• sudo –u <squid/tomcat user> crontab –l• Find there when logrotate should work and sizes are allowed

8 June 2011

frontier rpms 12

Testing the installation: load

(After a while,) monitor the upgraded server/s host load, verifying that it did not rise or drop considerably, as a result of the upgrade:

• CERN: use lemon https://lemonweb.cern.ch/

• Else where: ganglia or whatever host monitoring tool is in use

8 June 2011

frontier rpms 13

Building a frontier rpm

• rpms should be built at voatlas137 • The related sources should be checked out

from svn, under: /data/$USER/rpms 1) Build an rpm:

cd /data/$USER/rpms/<rpm Name> pushd ./scripts; ../../scripts/rpmbuild_.sh;popd

2) Copy rpm to the web: cd /data/$USER/rpms/<rpm Name> pushd ./scripts; ../../scripts/rpm2web.sh ; popd

8 June 2011

frontier rpms 14

Plan

1) Once the new rpms will be defined as production ready,it will be suggested that all ATLAS frontier sites will use these frontier rpm/s instead of existing rpm or tarball/s

2) Enhance frontier rpms to support CMS- Support more than one servlet at a site.

ATLAS does not require this because it has only one servlet per site- Add a /etc/tomcat/servlets.conf configuration file- Add a /etc/tomcat/servlets.passwd configuration file

(will reside at sindes at CERN)

3) Avoid file duplication at underlying (CMS) tarball and at the rpms code (mainly at frontier-squid). This will make the maintenance of frontier code simpler

4) TBD: Add more monitoring tools into the rpms

8 June 2011

frontier rpms 158 June 2011

frontier rpms 16

Configuration: frontier-squidOptional configuration files: 1) /etc/squid/squidconf:

Used at rpm installationConsists of export commands Main usage: to specify user and group for frontier-squid, other than default user and group: squid, squid Example (as of CERN): export FRONTIER_USER=dbfrontier export FRONTIER_GROUP=users

2) /etc/squid/customize.sh Used per restart of the squid server. Consists of specific commands that control the content of squid.conf file.Examples: Appear at /etc/squid/customize.sh.example_* and at https://svnweb.cern.ch/trac/frontier/browser/rpms/frontier-squid/trunk/src/frontier-squid-utils/etc/squid

-See also: Flavia Donno's documentation for previous version of frontier-squid rpms: https://twiki.cern.ch/twiki/bin/view/PDBService/SquidRPMsTier1andTier2

8 June 2011

frontier rpms 17

Configuration: frontier-servlet

Per servlet configuration file should be edited manually. May be done after installation: <install dir>/tomcat/conf/Catalina/localhost/<servlet name>.xml

Example of such a file name: /data/dbfrontier/tomcat/conf/Catalina/localhost/atlr.xml

8 June 2011

frontier rpms 18

Configuration: frontier-awstatsOptional file to control environment variables at install time

/etc/awstats/awstats.confSyntax: export <variable>=<value>Example: export FRONTIER_USER=<something other than dbfrontier>

Editing of the following files may be done (before or) after rpm installation:1) A file for rsync with the password of user $SITE$PROJ

(Example: cernatlas) /etc/awstats/password-file

2) An optional per awstats client site/machine config file. For CERN clients, this file is not needed, but for other sites it may be needed. File: /etc/awstats/customize.sh ## WARNINGCare should be taken when changing variable in this file,( and in particular PATH_LOG). Testing of frontier rpms has not been done with changing of these variables

The rpm installs an example file: /etc/awstats/customize.sh.example_CERN: https://svnweb.cern.ch/trac/frontier/browser/rpms/frontier-awstats/trunk/src/etc/awstats/customize.sh.example_CERN

8 June 2011

frontier rpms 19

urls1. Documentation of new versions of frontier rpms:

http://frontier.cern.ch/dist/rpms/_README

2. The main url of frontier: http://frontier.cern.ch

3. Current frontier rpms: http://frontier.cern.ch/dist/rpms/

4. To browse svn files of frontier rpms: https://svnweb.cern.ch/trac/frontier/browser/rpms

5. To checkout a frontier rpms svn directory (example: frontier-tomcat): svn co svn+ssh://svn.cern.ch/reps/frontier/rpms/frontier-tomcat/trunc frontier-tomcat

6. Atlas frontier documentation by John Destefano: https://www.racf.bnl.gov/docs/services/frontier/

7. Documentation of (previous) frontier server rpms, by Flavia Donno:https://twiki.cern.ch/twiki/bin/view/LCG/WLCGFrontier

8. Documentation of (previous) versions of frontier-squid rpm, by Flavia Donno:https://twiki.cern.ch/twiki/bin/view/PDBService/SquidRPMsTier1andTier2

9. Previous versions of frontier rpms, created by Flavia Donno, are at: http://grid-deployment.web.cern.ch/grid-deployment/flavia/?C=M;O=D

10. CMS has a lot of frontier documentation

8 June 2011