GNU Parallel și GNU Stow

GNU Parallel s, i GNU StowZiua Libertat, ii Programelor s, i 30 de ani de GNU ın Balt, i

Radu Dumbraveanu

Universitatea de Stat “A. Russo” din Balt, i, Republica Moldova

Balt, i, 2013

GNU Parallel

GNU parallel este un program cu interfat, a ın linia de comanda(CLI) care permite utilizatorului executarea ın paralel a uneicomenzi sau a unui set de comenzi (script shell).

Sintaxa:

parallel [optiuni] [comanda [argumente]] < lista de argumente

sau

parallel [optiuni] [comanda [argumente]] ::: argumente

GNU Parallel

GNU parallel este un program cu interfat, a ın linia de comanda(CLI) care permite utilizatorului executarea ın paralel a uneicomenzi sau a unui set de comenzi (script shell).

Sintaxa:

parallel [optiuni] [comanda [argumente]] < lista de argumente

sau

parallel [optiuni] [comanda [argumente]] ::: argumente

Exemple clasice de utilizare

(echo ls; echo grep root /etc/passwd) | parallel

parallel traceroute ::: foss.org.my debian.org freenetproject.org

parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org

find . −type f | parallel gzip

ls ∗.gz | parallel −X mv {} archive

Exemple clasice de utilizare

(echo ls; echo grep root /etc/passwd) | parallel

parallel traceroute ::: foss.org.my debian.org freenetproject.org

parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org

find . −type f | parallel gzip

ls ∗.gz | parallel −X mv {} archive

Exemple clasice de utilizare

(echo ls; echo grep root /etc/passwd) | parallel

parallel traceroute ::: foss.org.my debian.org freenetproject.org

parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org

find . −type f | parallel gzip

ls ∗.gz | parallel −X mv {} archive

Exemple clasice de utilizare

(echo ls; echo grep root /etc/passwd) | parallel

parallel traceroute ::: foss.org.my debian.org freenetproject.org

parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org

find . −type f | parallel gzip

ls ∗.gz | parallel −X mv {} archive

Exemple clasice de utilizare

(echo ls; echo grep root /etc/passwd) | parallel

parallel traceroute ::: foss.org.my debian.org freenetproject.org

parallel −j0 ping −nc 3 ::: foss.org.my gnu.org freenetproject.org

find . −type f | parallel gzip

ls ∗.gz | parallel −X mv {} archive

Dificultat, i?

Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:

The program ’parallel’ is currently not installed. You caninstall it by typing:sudo apt−get install moreutils

Nu urmat, i acesta indicat, ie; comanda parallel din pachetulmoreutils este cu totul altceva.

Dificultat, i?

Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:

The program ’parallel’ is currently not installed. You caninstall it by typing:sudo apt−get install moreutils

Nu urmat, i acesta indicat, ie; comanda parallel din pachetulmoreutils este cu totul altceva.

Dificultat, i?

Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:

The program ’parallel’ is currently not installed. You caninstall it by typing:sudo apt−get install moreutils

Nu urmat, i acesta indicat, ie; comanda parallel din pachetulmoreutils este cu totul altceva.

Modalitat, i de instalare

Putem instala parallel ın felul urmator:

1 sudo apt−get install parallel

sau

1 wget ftp://ftp.gnu.org/gnu/parallel/parallel−20130922.tar.bz22 tar xf parallel−20130922.tar.bz23 cd parallel−201309224 ./configure5 make6 make install

20130922 = versiunea parallel din 22.09.2013

Modalitat, i de instalare

Putem instala parallel ın felul urmator:

1 sudo apt−get install parallel

sau

1 wget ftp://ftp.gnu.org/gnu/parallel/parallel−20130922.tar.bz22 tar xf parallel−20130922.tar.bz23 cd parallel−201309224 ./configure5 make6 make install

20130922 = versiunea parallel din 22.09.2013

Modalitat, i de instalare

Putem instala parallel ın felul urmator:

1 sudo apt−get install parallel

sau

1 wget ftp://ftp.gnu.org/gnu/parallel/parallel−20130922.tar.bz22 tar xf parallel−20130922.tar.bz23 cd parallel−201309224 ./configure5 make6 make install

20130922 = versiunea parallel din 22.09.2013

Modalitat, i de instalare

Daca nu sıntet, i root atunci:creat, i dosarele ~/bin s, i ~/share apoi

1 wget ftp://ftp.gnu.org/gnu/parallel/parallel−20130922.tar.bz22 tar xf parallel−20130922.tar.bz23 cd parallel−201309224 ./configure −−prefix=$HOME5 make6 make install

Modalitat, i de instalare

Sau daca dorit, i sa utilizat, i parallel fara a-l instala (de exemplupentru a experimenta):

1 wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel2 chmod 755 parallel3 cp parallel sem4 mv parallel sem ˜/bin/

parallel este un script Perl, nu ai ce compila.Mai multa informat, ie pehttps://github.com/razrichter/gnu_parallel.

Modalitat, i de instalare

Sau daca dorit, i sa utilizat, i parallel fara a-l instala (de exemplupentru a experimenta):

1 wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel2 chmod 755 parallel3 cp parallel sem4 mv parallel sem ˜/bin/

parallel este un script Perl, nu ai ce compila.Mai multa informat, ie pehttps://github.com/razrichter/gnu_parallel.

Dificultat, i?

Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:

1 /bin/bash: {}: command not found

sau

1 /bin/bash: :::: command not found

Solut, ii:

I adaugat, i opt, iunea --gnu (parallel --gnu ...);

I sau sudo rm /etc/parallael/config;

I sau http://stackoverflow.com/questions/16448887/

gnu-parallel-not-working-at-all

Dificultat, i?

Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:

1 /bin/bash: {}: command not found

sau

1 /bin/bash: :::: command not found

Solut, ii:

I adaugat, i opt, iunea --gnu (parallel --gnu ...);

I sau sudo rm /etc/parallael/config;

I sau http://stackoverflow.com/questions/16448887/

gnu-parallel-not-working-at-all

Dificultat, i?

Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:

1 /bin/bash: {}: command not found

sau

1 /bin/bash: :::: command not found

Solut, ii:

I adaugat, i opt, iunea --gnu (parallel --gnu ...);

I sau sudo rm /etc/parallael/config;

I sau http://stackoverflow.com/questions/16448887/

gnu-parallel-not-working-at-all

Dificultat, i?

Lansarea oricarei comenzi ce cont, ine parallel se sfırs, es, te cumesajul de eroare:

1 /bin/bash: {}: command not found

sau

1 /bin/bash: :::: command not found

Solut, ii:

I adaugat, i opt, iunea --gnu (parallel --gnu ...);

I sau sudo rm /etc/parallael/config;

I sau http://stackoverflow.com/questions/16448887/

gnu-parallel-not-working-at-all

Exemple de utilizare (divizarea fis, ierelor mari pe blocuri)

cat fisiermare.sql | parallel −k −−pipe grep −i ”text”

cat bigfiledump.sql | parallel −k −−pipe −−block 2M \grep −i ”text”

cat bigfiledump.sql | parallel −k −−pipe −−block 2M \−−sshlogin server.example.com,server2.example.net \grep −i ”text”

Exemple de utilizare (divizarea fis, ierelor mari pe blocuri)

cat fisiermare.sql | parallel −k −−pipe grep −i ”text”

cat bigfiledump.sql | parallel −k −−pipe −−block 2M \grep −i ”text”

cat bigfiledump.sql | parallel −k −−pipe −−block 2M \−−sshlogin server.example.com,server2.example.net \grep −i ”text”

Modalitat, i de transmitere a parametrilor

Cum transmitem parametrii la comanda care trebuie executata ınparalel?

1) Prin intrarea standard:

1 seq 5 | parallel echo

2 13 24 35 46 5

Modalitat, i de transmitere a parametrilor

Cum transmitem parametrii la comanda care trebuie executata ınparalel?1) Prin intrarea standard:

1 seq 5 | parallel echo

2 13 24 35 46 5

Modalitat, i de transmitere a parametrilor

Accesul la parametri:

1 seq 5 | parallel echo ”parm={}”

2 param=13 param=24 param=35 param=46 param=5

Modalitat, i de transmitere a parametrilor

Mai mult, i parametri deodata:

1 seq 10 | parallel −X echo

2 1 2 3 4 53 6 7 8 9 10

Modalitat, i de transmitere a parametrilor

2) Prin fis, iere:

1 parallel wget < adresse.lst

1 cat adrese.lst

2 foss.org.my3 debian.org4 freenetproject.org

Modalitat, i de transmitere a parametrilor

3) Prin linia de comanda:

1 parallel echo ::: 1 2 3 4

1 12 23 34 4

sau

1 parallel unzip ::: ./∗.zip

Modalitat, i de transmitere a parametrilor

3) Prin linia de comanda:

1 parallel echo ::: 1 2 3 4

1 12 23 34 4

sau

1 parallel unzip ::: ./∗.zip

Specificarea numarul de procese paralele

1) Numarul de procese = numarul de procesoare:

1 seq 10 20 | parallel −j+0 sleep

1 ps f −o pid,command

2 PID COMMAND3 31379 −bash4 32270 \ ps f −o pid,command5 30463 −bash6 30579 \ mc7 30581 \ bash −rcfile .bashrc8 32239 \ /usr/bin/perl −w /usr/bin/parallel −j+0 sleep9 32268 \ sleep 10

10 32269 \ sleep 11

Specificarea numarul de procese paralele

2) Numarul maximal de procese, explicit:

1 seq 10 20 | parallel −j 5 sleep

1 ps f −o pid,command

2 PID COMMAND3 31379 −bash4 32523 \ ps f −o pid,command5 30463 −bash6 30579 \ mc7 30581 \ bash −rcfile .bashrc8 32487 \ /usr/bin/perl −w /usr/bin/parallel −j 5 sleep9 32518 \ sleep 10

10 32519 \ sleep 1111 32520 \ sleep 1212 32521 \ sleep 1313 32522 \ sleep 14

Specificarea numarul de procese paralele

3) Numarul de sarcini per procesor, explicit:

1 $ seq 10 20 | parallel −j 150% sleep

1 ps f −o pid,command

2 PID COMMAND3 31379 −bash4 32636 \ ps f −o pid,command5 30463 −bash6 30579 \ mc7 30581 \ bash −rcfile .bashrc8 32602 \ /usr/bin/perl −w /usr/bin/parallel −j 150% sleep9 32632 \ sleep 10

10 32633 \ sleep 1111 32634 \ sleep 12

Contraexemplu

1 cd dosarcu66925fisiere2 /usr/bin/time cp −r ∗ /tmp

3 1.77user 49.62system 6:49.01elapsed 12%CPU4 7692784inputs+7367600outputs (0major+320minor)

1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j+0 cp −r {} /tmp

3 2.40user 37.61system 6:30.38elapsed 10%CPU4 7402904inputs+7367600outputs (21major+27017minor)

1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j 150% cp −r {} /tmp

3 2.51user 50.74system 12:55.95elapsed 6%CPU4 7579504inputs+7367608outputs (27major+28524minor)

Contraexemplu

1 cd dosarcu66925fisiere2 /usr/bin/time cp −r ∗ /tmp

3 1.77user 49.62system 6:49.01elapsed 12%CPU4 7692784inputs+7367600outputs (0major+320minor)

1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j+0 cp −r {} /tmp

3 2.40user 37.61system 6:30.38elapsed 10%CPU4 7402904inputs+7367600outputs (21major+27017minor)

1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j 150% cp −r {} /tmp

3 2.51user 50.74system 12:55.95elapsed 6%CPU4 7579504inputs+7367608outputs (27major+28524minor)

Contraexemplu

1 cd dosarcu66925fisiere2 /usr/bin/time cp −r ∗ /tmp

3 1.77user 49.62system 6:49.01elapsed 12%CPU4 7692784inputs+7367600outputs (0major+320minor)

1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j+0 cp −r {} /tmp

3 2.40user 37.61system 6:30.38elapsed 10%CPU4 7402904inputs+7367600outputs (21major+27017minor)

1 cd dosarcu66925fisiere2 ls . | /usr/bin/time parallel −j 150% cp −r {} /tmp

3 2.51user 50.74system 12:55.95elapsed 6%CPU4 7579504inputs+7367608outputs (27major+28524minor)

Adrese web

Pagina web

http://www.gnu.org/software/parallel

Manual s, i exemple

http://www.gnu.org/software/parallel/man.html

Surselehttp://savannah.gnu.org/git/?group=parallel

Adrese web

Tutorialehttp://savannah.gnu.org/projects/parallel

Video (partea I s, i II)

http://www.youtube.com/watch?v=OpaiGYxkSuQ

http://www.youtube.com/watch?v=P40akGWJ_gY

Miscelaneuhttp://static.usenix.org/publications/login/2011-02/pdfs/Tange.pdf

http://wiki.scinethpc.ca/wiki/images/7/7b/Tech-talk-gnu-parallel.pdf

http://www.luga.de/Angebote/Vortraege/GNU_Parallel_LIT_2011/GNU_

Parallel_LIT_2011.pdf

http://www.admin-magazine.com/HPC/Articles/

GNU-Parallel-Multicore-at-the-Command-Line-with-GNU-Parallel

Adrese web

Comparat, ie cu alte comenzi (xargs, find -exec, make -j etc)

http://www.gnu.org/software/parallel/man.html#summary_table

GNU Stow

GNU Stow este un caz particular de manager de pachete.

Instalare:

1 sudo apt−get install stow

Instalarea pachetelor

Cum instalam un pachet cu ajutorul lui GNU Stow?

1 sudo make install prefix=/usr/local/stow/numepachet2 cd /usr/local/stow3 sudo stow numepachet

Pachetele care nu folosesc GNU AutoConf pot sa nu accepteparametrul “prefix”, atunci este nevoie de modificari ın fis, ierulMakefile.

Instalarea pachetelor

Cum instalam un pachet cu ajutorul lui GNU Stow?

1 sudo make install prefix=/usr/local/stow/numepachet2 cd /usr/local/stow3 sudo stow numepachet

Pachetele care nu folosesc GNU AutoConf pot sa nu accepteparametrul “prefix”, atunci este nevoie de modificari ın fis, ierulMakefile.

Eliminarea pachetelor

1 cd /usr/local/stow/2 sudo stow −D numepachet

Exemplu: mai multe versiuni ale unuia s, i aceluias, i pachetın aceeas, i distribut, ie

Mai multe versiuni ale unuia s, i aceluias, i pachet ımpreuna.

1 cd numepachet−3.02 sudo make install prefix=/usr/local/stow/numepachet−3.0

3 cd numepachet−3.1−beta4 sudo make install prefix=/usr/local/stow/numepachet−3.1−beta

Exemplu: mai multe versiuni ale unuia s, i aceluias, i pachetın aceeas, i distribut, ie

Putem accesa fis, iere ambelor pachete prin

1 /usr/local/stow/numepachet−3.0/bin/fisierbinar

s, i

2 /usr/local/stow/numepachet−3.1−beta/bin/fisierbinar

Iar daca executam

1 cd /usr/local/stow2 sudo stow numepachet−3.0

atunci lansınd ın linia de comanda doar fisierbinar vom accesa/usr/local/stow/numepachet-3.0/bin/fisierbinar.

Adrese web

Pagina web

http://www.gnu.org/software/stow/

Manual s, i exemple

http://www.gnu.org/software/stow/manual/