Bundling Packages and Deploying Applications with RPM

Bundling Packages and Deploying Applications

with RPM

Alexander Shopov <[email protected]>

[ash@edge ~]$ whoami

By day: Software Engineer at Cisco {,Nov. 28}By night: OSS contributorCoordinator of Bulgarian Gnome TP Contacts:

E-mail: [email protected] Jabber: [email protected] LinkedIn: http://www.linkedin.com/in/alshopov SlideShare: http://www.slideshare.net/al_shopovGitHub: https://github.com/alshopovWeb: Just search “al_shopov”

mailto:[email protected]

xmpp:[email protected]

http://www.linkedin.com/in/alshopov

http://www.slideshare.net/al_shopov

https://github.com/alshopov

http://lmgtfy.com/?q=al_shopov

Please Learn And Share

License: Creative Commons Attribution 4.0 InternationalCC-BY v4.0

http://creativecommons.org/licenses/by/4.0/

Why Package at All?

NICE & NEAT

As Number of Components Grows Complexity (Interactions) Grows as

Power of 2

As Years of Maintenence Grow Complexity Grows at Least Linearly

As Both Number of Components and Years Grow – Complexity

Explodes

Why RPM

● Widely used packaging format;● Used to mean: “Red Hat Package Manager”, now

it is “RPM Package Manager”;● Popular with Enterprise-ish distributions:

– Fedora;– Red Hat;

– CentOS;– Oracle's Linux;

– SUSE.

● The other popular format in Linux is “deb” used in Debian/Ubuntu.

Where Does RPM Stand?

Inno, NullSoft, InstallShiled, Wix, Windows Installer,

Installer for Mac OS

Make, imake, auto-tools, project/solution files, etc.

CompilingSource to Binary

PackagingBinary to Packages

Where Does RPM Stand?

Inno, NullSoft, InstallShiled, Wix, Windows Installer,

Installer for Mac OS

Make, imake, auto-tools, project/solution files, etc.

CompilingSource to Binary

PackagingBinary to Packages

RPM

The Packaging Process

SOURCES

BUILD

BUILDROOT

RPMS

SPECS

The Packaging Process

SOURCES

BUILD

BUILDROOT

RPMS

SPECS

%prep

%install

%files

● Uncompress & patch– rpmbuild -bp

● Configure/compile– rpmbuild -bc

● Install to a virtual file system – place, copy– rpmbuild -bi

● List files to be packaged– rpmbuild -bb/-bs/-ba

Structure of RPM file

● Lead/identifier/magic ( starts with abed dbee - hex)

● Signature - optional

● Header – tagged structure – tags/type/number/size/data for them

● Payload – archive. gzipped cpio (1977 by Dick Haight UNIX 1.0) → gzipped tar (UNIX 7th ed. – 1979)

Payload

Iden

tifi

er

Sig

nat

ure

Hea

der

Structure of RPM Package Name

● name-version-release.architecture.rpm● gzip-1.3.12-19.el6_4.x86_64.rpm● Name – what package is or does and purpose● Version – upstream version● Release – version of RPM (prefix for distro)● Architecture of the rpm

– noarch

– src

Common and Rarer ArchiecturesSuffix Architecture Fedora RedHat CentOS Oracle

i686 32bit Intel x x x x

x86_64 64bit AMD x x x x

ia64 64bit Intel x x x

s390 24/31/32bit G4/5/6 x x

s390x IBM System Z – 64bit z900 x x

ppc 32bit PowerPC x x

ppc64 64bit PowerPC BE x x

ppc64le 64bit PowerPC LE (

arm 32bit ARM5-7 )

armhfp 32bit ARM7 x

aaarch64 64bit ARM8+ x

Important Packages to Install

● redhat-rpm-config● rpm-build● rpmdevtools● rpmlint● Other if you want to contribute to

Fedora/Redhat/CentOS. Oracle have not published for them

redhat-rpm-config

● /etc/rpm/macros* - others also install here - macros here perl,python, nodejs, java

● /usr/lib/rpm/redhat – macros and shell scripts

rpm-build

● rpmbuild – command to actually build● rpmspec – query spec files● gendiff – diff wrapper

rpmlint

● rpmlint – check errors in specs and packages● rpmdiff – check differences between packages

rpmdevtools

● rpmfile, rpmls, rpmpeek, rpmelfsym, rpmargs● /etc/rpmdevtools – spec templates● rpmdev-bumpspec – auto increment and

maintain spec files● rpmdev-setuptree – create the spec tree● rpmdev-newinit, rpmdev-newspec

Let's Build A Package!

● Let's build Tomcat 8 package● Proper ownership of files● Add a startup/shtdown script● Deploy a small package in Tomcat● Make the package take the place of ROOT

context● Switch provider of JPA – from Derby to Oracle

RDBMS

Prepare The Build Environment

● yum install rpmdevtools● Dependencies:

– fakeroot

– fakeroot-libs

– rpm-build

DON'T BUILD AS ROOT!

ROOT

FAKE IT!

LEAVE IT TO FAKEROOT

Create Designated User for Building of RPM

● useradd packager (will make a user group with same name)

● passwd packager● Do not add designated user to groups:

– root/wheel

– adm/sys/shutdown/sshd, others

– oinstall/dba/oper (if you have Oracle RDBMS on the same machine for some weird reason)

Create RPM Building Environment

● rpmdev-setuptree

● ~/.rpmmacros

– %_topdir %(echo $HOME)/rpmbuild

● ls ~/rpmbuild/

– BUILD RPMS SOURCES SPECS SRPMS

– mkdir ~/rpmbuild/BUILDROOT

● rpm --eval '%buildroot'

– /home/packager/rpmbuild/BUILDROOT/%{name}-%{version}-%{release}.i386

rpmdev-newspec -o ~/rpmbuild/SPECS/counterbean.spec

Starting the Build

preamble

%prep

%build

%install

%clean

%files

%changelog

Name: counterbeanVersion: Release: 1%{?dist}Summary:

Group: License: URL: Source0:

BuildRequires: Requires:

%description

%prep%setup -q

%build%configuremake %{?_smp_mflags}

%installrm -rf $RPM_BUILD_ROOTmake install DESTDIR=$RPM_BUILD_ROOT

%cleanrm -rf $RPM_BUILD_ROOT

%files%defattr(-,root,root,-)%doc

%changelog

Fix the Preamble

Name: counterbeanVersion: 2.0.1Release: 1%{?dist}Summary: Keep a keen eye on beans for counting purposes# one of '/usr/share/doc/rpm-*/GROUPS'Group: Applications/ProductivityLicense: ASL 2.0URL: https://github.com/alshopov/counterbeanSource0: apache-tomcat-8.0.14.tar.gz

# BuildRequires: TODO# Requires: TODO

# no need to set BuildRoot/٪buildroot/$RPM_BUILD_ROOT # it is set for us# BuildRoot:

Fix the Preamble




Comments start with “#”Place comments on lines

Fix the Preamble




The Voldemort Character –Do not name “%”

Fix the Preamble




We will fix these later

%define prefix /opt/%{name}

%descriptionA sample Java application for packaging demo purposes. You can also use it to count: * beans; * kogs; * minions.

%prep# -q: no output, -c: create directory%setup -q -c

%installinstall -m 0755 -d ${RPM_BUILD_ROOT}%{prefix}cp -r apache-tomcat-8.0.14/* ${RPM_BUILD_ROOT}%{prefix} %cleanrm -fr ${RPM_BUILD_ROOT}

%files%defattr(-,root,root,-)/opt/counterbean%doc

%changelog* Mon Oct 13 2014 Alexander Shopov <[email protected]>- Initial version containing only tomcat.

[root@localhost ashopov]# yum install \ ~packager/rpmbuild/RPMS/i386/counterbean-2.0.1-1.el6.i386.rpmLoaded plugins: refresh-packagekit, securitySetting up Install ProcessExamining /home/packager/rpmbuild/RPMS/i386/counterbean-2.0.1-1.el6.i386.rpm: counterbean-2.0.1-1.el6.i386Marking /home/packager/rpmbuild/RPMS/i386/counterbean-2.0.1-1.el6.i386.rpm to be installedResolving Dependencies--> Running transaction check---> Package counterbean.i386 0:2.0.1-1.el6 will be installed--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================== Package Arch Version Repository Size======================================================================================Installing: counterbean i386 2.0.1-1.el6 /counterbean-2.0.1-1.el6.i386 12 M

Transaction Summary======================================================================================Install 1 Package(s)

Total size: 12 MInstalled size: 12 MIs this ok [y/N]: yDownloading Packages:Running rpm_check_debugRunning Transaction TestTransaction Test SucceededRunning Transaction Installing : counterbean-2.0.1-1.el6.i386 1/1 Verifying : counterbean-2.0.1-1.el6.i386 1/1

Installed: counterbean.i386 0:2.0.1-1.el6

Complete!

[packager@localhost ~]$ /opt/counterbean/bin/startup.shNeither the JAVA_HOME nor the JRE_HOME environment variable is definedAt least one of these environment variable is needed to run this program[packager@localhost ~]$

Try To Start

We Forgot Dependencies

[packager@localhost ~]$ yum provides jreLoaded plugins: refresh-packagekit, security⋮⋮⋮1:java-1.6.0-openjdk-1.6.0.0-5.1.13.3.el6_5.i686 : OpenJDK Runtime EnvironmentRepo : public_ol6_latestMatched from:Other : jre⋮⋮⋮1:java-1.7.0-openjdk-1.7.0.51-2.4.4.1.0.1.el6_5.i686 : OpenJDK Runtime EnvironmentRepo : public_ol6_latestMatched from:Other : jre⋮⋮⋮

[root@localhost ~]# yum provides '*/java'Loaded plugins: refresh-packagekit, security⋮⋮⋮graphviz-java-2.26.0-7.el6.i686 : Java extension for graphvizRepo : public_ol6_latestMatched from:Filename : /usr/lib/graphviz/java⋮⋮⋮1:java-1.6.0-openjdk-1.6.0.0-6.1.13.4.el6_5.i686 : OpenJDK Runtime EnvironmentRepo : public_ol6_latestMatched from:Filename : /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/bin/java⋮⋮⋮

Add Dependencies

Release: 2%{?dist}Requires: jre

%changelog* Mon Oct 13 2014 Alexander Shopov <[email protected]> Add jre as runtime dependency

* Mon Oct 13 2014 Alexander Shopov <[email protected]> Initial version containing only tomcat.

[root@localhost ashopov]# yum install \ ~packager/rpmbuild/RPMS/i386/counterbean-2.0.1-2.el6.i386.rpm Loaded plugins: refresh-packagekit, securitySetting up Install ProcessExamining /home/packager/rpmbuild/RPMS/i386/counterbean-2.0.1-2.el6.i386.rpm: counterbean-2.0.1-2.el6.i386Marking /home/packager/rpmbuild/RPMS/i386/counterbean-2.0.1-2.el6.i386.rpm as an update to counterbean-2.0.1-1.el6.i386Resolving Dependencies--> Running transaction check---> Package counterbean.i386 0:2.0.1-1.el6 will be updated---> Package counterbean.i386 0:2.0.1-2.el6 will be an update--> Processing Dependency: jre for package: counterbean-2.0.1-2.el6.i386--> Running transaction check---> Package java-1.7.0-openjdk.i686 1:1.7.0.65-2.5.1.2.0.1.el6_5 will be installed--> Finished Dependency Resolution

Dependencies Resolved

Name: counterbeanVersion: 2.0.1Release: 3%{?dist}Summary: Keep a keen eye on beans for counting purposesGroup: Applications/ProductivityLicense: ASL 2.0URL: https://github.com/alshopov/counterbeanRequires: jre

%define tc_ver 8.0.14Source0: apache-tomcat-%{tc_ver}.tar.gz

%description…

%define prefix /opt/%{name}

%prep…

%installinstall -m 0755 -d ${RPM_BUILD_ROOT}%{prefix}cp -r apache-tomcat-%{tc_ver}/* ${RPM_BUILD_ROOT}%{prefix} %clean…

%files…

%changelog…

Parametrize the SPEC

Root Owns Everything – Let's Fix

Release: 4%{?dist}

%files%defattr(-,%{name},%{name},-)/opt/%{name}

%changelog* Tue Oct 14 2014 Alexander Shopov <[email protected]> Make 'counterbean' user own everything…

But When We Install…

warning: user counterbean does not exist - using rootwarning: group counterbean does not exist - using root

We Must First Create The User

Release: 5%{?dist}Requires(pre): shadow-utils

%pregetent group %{name} >/dev/null || groupadd -r %{name}getent passwd %{name} >/dev/null || \ useradd -r -g %{name} -d %{prefix} -s /sbin/nologin \ -c "Account to own and run %{name}" %{name}exit 0

%changelog* Tue Oct 14 2014 Alexander Shopov <[email protected]> Create user and group for 'counterbean'…

Create Template

[packager@localhost rpmbuild]$ rpmdev-newinit --help⋮⋮⋮The template used is /etc/rpmdevtools/template.init.

[packager@localhost rpmbuild]$ rpmdev-newinit \ /rpmbuild/SOURCES/counterbean.initSkeleton init script has been created to "/home/packager/rpmbuild/SOURCES/counterbean.init".# The template used is /etc/rpmdevtools/template.init.

#!/bin/sh## counterbean - <summary>## chkconfig: <default runlevel(s)> <start> <stop># description: <description, split multiple lines\# with a backslash># http://fedoraproject.org/wiki/FCNewInit/Initscripts### BEGIN INIT INFO# Provides: # Required-Start: # Required-Stop: # Should-Start: # Should-Stop: # Default-Start: # Default-Stop: # Short-Description: # Description: ### END INIT INFO

# Source function library.. /etc/rc.d/init.d/functions

exec="/usr/sbin/counterbean"prog=$(basename $exec)

[ -e /etc/sysconfig/$prog ] && \ . /etc/sysconfig/$prog

lockfile=/var/lock/subsys/$prog

start() {

echo -n $"Starting $prog: " # if not running, start it up here, usually something like "daemon $exec" retval=$? echo [ $retval -eq 0 ] && touch $lockfile return $retval}

Template Contents

stop() {

echo -n $"Stopping $prog: " # stop it here, often "killproc $prog" retval=$? echo [ $retval -eq 0 ] && rm -f $lockfile return $retval}

restart() {

stop start}

case "$1" in start|stop|restart) $1 ;; force-reload) restart ;; status) status $prog ;; try-restart|condrestart) if status $prog >/dev/null ; then restart fi

;; reload) # If config can be reloaded without restarting, implement it here, # remove the "exit", and add "reload" to the usage message below. # For example: # status $prog >/dev/null || exit 7 # killproc $prog -HUP action $"Service ${0##*/} does not support the reload action: " /bin/false exit 3 ;; *) echo $"Usage: $0 {start|stop|status|restart|try-restart|force-reload}" exit 2esac

#!/bin/sh## counterbean - count bean, kogs, minions## chkconfig: 345 98 02# description: small demo RDBMS Java web app

# Source function library.. /etc/rc.d/init.d/functions

start_cmd="runuser counterbean -g counterbean -s\ /bin/sh -c '/opt/counterbean/bin/startup.sh' "stop_cmd="runuser counterbean -g counterbean -s\ /bin/sh -c '/opt/counterbean/bin/shutdown.sh' "prog=counterbean

lockfile=/var/lock/subsys/$prog

start() { echo -n $"Starting $prog: " $start_cmd retval=$? echo [ $retval -eq 0 ] && touch $lockfile return $retval}

Startup Script

stop() { echo -n $"Stopping $prog: " $stop_cmd retval=$? echo [ $retval -eq 0 ] && rm -f $lockfile return $retval}

restart() { stop start}

case "$1" in start|stop|restart) $1 ;; status) status $prog ;; *) echo $"Usage: $0 {start|stop|restart|status}" exit 2esac

Use The ScriptRelease: 6%{?dist}Requires: coreutils, jre, chkconfig, initscriptsRequires(pre): shadow-utils

%define tc_ver 8.0.14Source0: apache-tomcat-%{tc_ver}.tar.gzSource1: %{name}.init

%installinstall -m 0755 -d ${RPM_BUILD_ROOT}%{prefix}cp -r apache-tomcat-%{tc_ver}/* ${RPM_BUILD_ROOT}%{prefix}install %{SOURCE1} -D ${RPM_BUILD_ROOT}/etc/rc.d/init.d/%{name}

%defattr(-,%{name},%{name},-)/opt/%{name}%attr(-,root,root) /etc/rc.d/init.d/%{name}

%changelog* Wed Oct 15 2014 Alexander Shopov <[email protected]> Proper startup scripts…

Use The Script

%pregetent group %{name} >/dev/null || groupadd -r %{name}getent passwd %{name} >/dev/null || \ useradd -r -g %{name} -d %{prefix} -s /sbin/nologin \ -c "Account to own and run %{name}" %{name}exit 0

%post/sbin/chkconfig --add %{name}if [ "$1" = 1 ]; then/sbin/service %{name} start > /dev/null 2>&1fiexit 0

%preunif [ "$1" = 0 ]; then/sbin/service %{name} stop > /dev/null 2>&1fiexit 0

%postunif [ "$1" -ge 1 ]; then/sbin/service %{name} restart > /dev/null 2>&1fiexit 0

%pre, %post, %preun, %postun?

● Scripts that get executed on install, upgrade, remove;

● There are others:– %triggerin, %triggerun, %triggerpostun:

ADVANCED – dependencies on packages;

– %pretrans, %posttrans: VERY ADVANCED, avoid (will fail in chroot, initial install, usually written in Lua.

Execution of Scripts

install upgrade uninstall

%pre $1 == 1 $1 == 2 Not run

%post $1 == 1 $1 == 2 Not run

%preun Not run $1 == 1 $1 == 0

%postun Not run $1 == 1 $1 == 0

1 2 3 4 5 6

%pre of new package

package install

%post of new package

%preun of old package

removal of old package

%postun of old package

$1 == 2 $1 == 2 $1 == 1 $1 == 1

Introducing CounterBean – Enterprise version

Very Modern App

Very Modern App

HTML5

Bootstrap 3.2

Very Modern App

Servlet 3.0No web.xml

Servlet 3.0Annotations

Very Modern App

In-Memory Derby

JPA 2.1With EclipseLink 2.5.x

When Deploying It

● Remove default applications from Tomcat● Make it the ROOT application

– Expand as ROOT in Tomcat's webapps folder

– Remove context.xml

● Migrate to Oracle RDBMS– Fix persistence.xml

– Add Oracle's driver

Make Counterbean the Default App

Release: 7%{?dist}

%define tc_ver 8.0.14%define cb_ver 1.0-SNAPSHOTSource0: apache-tomcat-%{tc_ver}.tar.gzSource1: %{name}-%{cb_ver}.warSource2: %{name}.init

%installinstall -m 0755 -d ${RPM_BUILD_ROOT}%{prefix} cp -r apache-tomcat-%{tc_ver}/* ${RPM_BUILD_ROOT}%{prefix}# Remove default applicationsrm -fr ${RPM_BUILD_ROOT}%{prefix}/webapps# Ensure ROOT exists and unzip counterbean to itinstall -m 0755 -d ${RPM_BUILD_ROOT}%{prefix}/webapps/ROOTunzip %{SOURCE1} -d ${RPM_BUILD_ROOT}%{prefix}/webapps/ROOT# Sets the context, we need the defaultrm -fr ${RPM_BUILD_ROOT}%{prefix}/webapps/ROOT/META-INF/context.xmlinstall %{SOURCE2} -D ${RPM_BUILD_ROOT}/etc/rc.d/init.d/%{name}

The DBA Prepares the RDBMS

SQL> CREATE USER counterbean IDENTIFIED BY counterbean;

User created.

SQL> GRANT RESOURCE, CONNECT TO counterbean;

Grant succeeded.

SQL> ALTER USER counterbean QUOTA 50M ON USERS;

User altered.

The DBA Prepares the RDBMS

SQL> CREATE USER counterbean IDENTIFIED BY counterbean;

User created.

SQL> GRANT RESOURCE, CONNECT TO counterbean;

Grant succeeded.

SQL> ALTER USER counterbean QUOTA 50M ON USERS;

User altered.

AKA “Mr. Generous DBA”

Use Oracle RDBMS

Release: 8%{?dist}

⋮⋮⋮Source2: %{name}.initSource3: ojdbc7.jarSource4: persistence.xml

%install⋮⋮⋮# Sets the context, we need the defaultrm -fr ${RPM_BUILD_ROOT}%{prefix}/webapps/ROOT/META-INF/context.xmlinstall %{SOURCE2} -D ${RPM_BUILD_ROOT}/etc/rc.d/init.d/%{name}

# Replace JDBC driverrm ${RPM_BUILD_ROOT}%{prefix}/webapps/ROOT/WEB-INF/lib/derby-*.jarcp %{SOURCE3} ${RPM_BUILD_ROOT}%{prefix}/webapps/ROOT/WEB-INF/lib/# Replace persistence.xml cp %{SOURCE4} ${RPM_BUILD_ROOT}%{prefix}\ /webapps/ROOT/WEB-INF/classes/META-INF/persistence.xml

Final File Touches

Release: 9%{?dist}

%files%defattr(-,%{name},%{name},-)/opt/%{name}%attr(-,root,root) /etc/rc.d/init.d/%{name}%doc /opt/%{name}/LICENSE%doc /opt/%{name}/NOTICE%doc /opt/%{name}/RELEASE-NOTES%doc /opt/%{name}/RUNNING.txt%attr(755,%{name},%{name}) %ghost /opt%{name}/conf/Catalina/%ghost /opt/%{name}/logs/%ghost /opt/%{name}/work/%config /opt/%{name}/webapps/ROOT/WEB-INF/classes/META-INF/persistence.xml

yum install createrepomkdir -p /somewhere/{SRPMS,i386,x86_64}# copy RPMs to the directoriesfor arch in /somewhere/*; do pushd ${arch} createrepo . popddone

Publishing RPMs As Yum Repositories

cat /etc/yum.repos.d/example.repo

[example]name=Example Repositorybaseurl=ftp://example.com/somewhere/$basearchenabled=1

Using the New Repository

rpm -ivh counterbean-2.0.1-10.el6.i386# Install, Verbose, Hashes show progressrpm -U *.rpm # Upgraderpm -F *.rpm # Freshen – install if existsrpm -qa | grep -i counterbean # Query Allrpm -qpR counterbean-2.0.1-10.el6.i386# Query what the package Provide and Requiresrpm -ql counterbean # Query List files in packagerpm -e counterbean # Eraserpm -qf /etc/passwd # Query for package of Filerpm -qi counterbean # infomration# do the query on a package rather than the RPM DBrpm -qip counterbean-2.0.1-10.el6.i386 # or -qilprpm -Vp # verify package – changes to filesrpm --rebuilddb # rebuild database

Popular RPM Commands

yum install counterbean # install, -y automaticallyyum remove counterbean # uninstallsyum update -y # auto upgrades allyum search counterbean # find such a packageyum info counterbean # info about a packageyum list # list packages # available|installed|extras|updates|obsoletesyum group[list|info|install|update|remove] # groupsyum provides /etc/passwdyum repolist # enables repositoriesyum repolist all # allyum --[enable|disable]repo=repo_id # enable/disable

Popular YUM Commands

Further Resources● Max RPM Book: http://www.rpm.org/max-rpm/

● GuruLabs RPM Guide: https://www.gurulabs.com/downloads/GURULABS-RPM-LAB/GURULABS-RPM-GUIDE-v1.0.PDF

● Fedora Project

– How to Create an RPM Package: https://fedoraproject.org/wiki/How_to_create_an_RPM_package

– Packaging Guidelines: https://fedoraproject.org/wiki/Packaging:Guidelines

● Clint Savage, How to Make an RPM: https://www.youtube.com/watch?v=4J_Iksu1fgo

● Jeff's Linux Vids: https://www.youtube.com/channel/UC96flr-XcwMTeyZOhGPfkLw

http://www.rpm.org/max-rpm/

https://www.gurulabs.com/downloads/GURULABS-RPM-LAB/GURULABS-RPM-GUIDE-v1.0.PDF

https://www.gurulabs.com/downloads/GURULABS-RPM-LAB/GURULABS-RPM-GUIDE-v1.0.PDF

https://fedoraproject.org/wiki/How_to_create_an_RPM_package

https://fedoraproject.org/wiki/Packaging:Guidelines

https://www.youtube.com/watch?v=4J_Iksu1fgo

https://www.youtube.com/channel/UC96flr-XcwMTeyZOhGPfkLw