Bash tricks

  • View
    333

  • Download
    1

Embed Size (px)

Text of Bash tricks

  • 3 bash one-liners

    Carlo zED Caputocarlo.caputo@gmail.com

  • 1

  • To l eu fazendo um dump de banco...

    $ mongoexport -d pseudo -c games | gzip > dump.json.gz

    connected to: 127.0.0.1

  • Demorou mais de 10 segundos...

    $ mongoexport -d pseudo -c games | gzip > dump.json.gz

    connected to: 127.0.0.1^C

  • T, preciso mostrar o progresso...

    $ mongoexport -d pseudo -c games | gzip | pv > dump.json.gz

    connected to: 127.0.0.17.83MB 0:00:05 [1.75MB/s] [ ]

  • Hunnn... Ainda no sei quanto falta.

    $ mongoexport -d pseudo -c games | pv -c -N raw | gzip | pv -c -N gz > dump.json.gz connected to: 127.0.0.1 raw: 54.5MB 0:00:07 [8.28MB/s] [ ] gz: 11.1MB 0:00:07 [1.72MB/s] [ ]

  • Ah, eu sei quantas linhas so...

    $ ( export H=localhost D=pseudo C=games; export F="dump-$H-$D-$C-$(date +%Y%m%d_%H%M%S)"; mongoexport -h $H -d $D -c $C 2> $F.log | pv -c -N raw | pv -c -N rows -l -s "$( echo -e "use $D;\ndb.$C.find().count();" | mongo $H 2>> $F.log | grep "switched to db $D" -A1 | tail -1 )" | gzip | pv -c -N gz | tee $F.gz | md5sum > $F.md5 )

    raw: 78.4MB 0:00:10 [7.97MB/s] [ ] gz: 16.1MB 0:00:10 [1.69MB/s] [ ] rows: 46.7k 0:00:10 [4.59k/s] [> ] 63% ETA 0:00:05

  • Mas t muito idle ainda....

    top - 23:19:46 up 53 days, 17 min, 24 users, load average: 0.15, 0.12, 0.09Tasks: 334 total, 2 running, 331 sleeping, 0 stopped, 1 zombieCpu0 : 33.3%us, 8.8%sy, 0.0%ni, 57.9%id, 0.0%wa, 0.0%hi, 0.0%si,Cpu1 : 2.3%us, 1.0%sy, 0.3%ni, 94.2%id, 2.3%wa, 0.0%hi, 0.0%si,Cpu2 : 85.2%us, 4.6%sy, 0.0%ni, 10.2%id, 0.0%wa, 0.0%hi, 0.0%si,Cpu3 : 2.6%us, 1.0%sy, 0.3%ni, 94.1%id, 2.0%wa, 0.0%hi, 0.0%si,Mem: 4107620k total, 3687764k used, 419856k free, 278800k buffersSwap: 4192960k total, 363512k used, 3829448k free, 658816k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24448 zed 20 0 17688 4300 2880 R 98 0.1 0:10.45 mongoexport 24451 zed 20 0 2052 504 344 S 31 0.0 0:03.20 gzip 24449 zed 20 0 4056 644 568 S 10 0.0 0:01.04 pv 24450 zed 20 0 4056 656 576 S 8 0.0 0:00.81 pv 3587 mongodb 20 0 804m 120m 119m S 1 3.0 0:08.59 mongod 24456 zed 20 0 3892 508 440 S 1 0.0 0:00.07 md5sum 24454 zed 20 0 3884 548 480 S 1 0.0 0:00.05 tee 24452 zed 20 0 4056 660 568 S 0 0.0 0:00.03 pv

  • Vamos ver...

    $ time --help--help: command not found

    $ type timetime is a shell keyword

    $ help timetime: time [-p] pipeline Report time consumed by pipeline's execution.

    $ /usr/bin/time --helpUsage: /usr/bin/time [-apvV] [-f format] [-o file] [--append] [--verbose] [--portability] [--format=format] [--output=file] [--version] [--quiet] [--help] command [arg...]

  • Ei, espera a, que tal usar xz?!

    $ mongoexport -d pseudo -c games | head -c 10000 | tee >( ( /usr/bin/time -f 'gz: %E real, %U user, %S sys' gzip -9 | wc -c | sed 's/^/gz: /' ) 1>&2) | /usr/bin/time -f 'xz: %E real, %U user, %S sys' xz -9 | wc -c | sed 's/^/xz: /'

    connected to: 127.0.0.1gz: 0:00.24 real, 0.00 user, 0.00 sysgz: 2078xz: 0:00.31 real, 0.02 user, 0.04 sysxz: 2012

  • Melhor limitar por tempo do que por bytes. Mas como parar?...$ { ( mongoexport -d pseudo -c games | tee >(wc -c | sed 's/^/raw: /' 1>&2) | tee >( ( /usr/bin/time -f 'gz: %E real, %U user, %S sys' gzip -9 | wc -c | sed 's/^/gz: /' ) 1>&2) | /usr/bin/time -f 'xz: %E real, %U user, %S sys' xz -9 | wc -c | sed 's/^/xz: /' ) & } ; sleep 9 ; kill -INT %% [1] 12289connected to: 127.0.0.1$ raw: 14623779gz: 0:08.99 real, 0.75 user, 0.00 sysgz: 2706591Command terminated by signal 2xz: 0:09.02 real, 8.78 user, 0.17 sys

  • Ahh, a est o nmero perdido!

    $ { ( mongoexport -d pseudo -c games | tee >(wc -c | sed 's/^/raw: /' 1>&2) | tee >( ( /usr/bin/time -f 'gz: %E real, %U user, %S sys' gzip -9 | wc -c | sed 's/^/gz: /' ) 1>&2) | /usr/bin/time -f 'xz: %E real, %U user, %S sys' xz -9 | wc -c | sed 's/^/xz: /' ) & } ; sleep 9 ; kill -INT $(pstree -p $! | sed -n 's/.*mongoexport(\([0-9]*\)).*/\1/p') [1] 12724connected to: 127.0.0.1$ raw: 14497402gz: 0:09.10 real, 0.73 user, 0.01 sysgz: 2695783xz: 0:09.18 real, 8.73 user, 0.16 sysxz: 2021928

  • Mas deixa eu formatar isso melhor...

    $ ( { ( { mongoexport -d pseudo -c games 2>/dev/null | tee >(wc -c | sed 's/^/s_raw=/' 1>&2) | tee >( ( /usr/bin/time -f 't_gz=%U+%S' gzip -9 | wc -c | sed 's/^/s_gz=/') 1>&2) | /usr/bin/time -f 't_xz=%U+%S' xz -9 | wc -c | sed 's/^/s_xz=/' ; echo 'scale=2; print "\nxz is ", t_xz/t_gz; r_sz=s_gz/s_xz; r_gz=s_gz/s_raw; r_xz=s_xz/s_raw; scale=0; print " times slower and ", (r_sz-1)/.01,"% smaller\ncompression ratios: gz=",(1-r_gz)/.01,"%, xz=",(1-r_xz)/.01,"%\n"' ; } 2>&1 | bc ) & } ; sleep 9 ; kill -INT $(pstree -p $! | sed -n 's/.*mongoexport(\([0-9]*\)).*/\1/p') 2>/dev/null ) xz is 12.02 times slower and 33% smallercompression ratios: gz=82%, xz=87%

  • Ai... :/ ser que tem algum outro nvel de compresso que valha a pena?

    $ ( { ( { { echo -n 'mongoexport -d pseudo -c games 2>/dev/null | tee' ; for z in 1 2 ; do for l in $(seq 0 9); do test $z = 2 -a $l = 0 || echo -n " >(/usr/bin/time -f 't[$z$l]=%U+%S' $(test $z == 2 && echo gzip || echo xz) -$l | wc -c | sed 's/^/s[$z$l]=/' 1>&2)" ; done ; done ; echo -n ' | cat > /dev/null' ; } | bash 2>&1 ; echo 'scale=4; for(p=1;p /dev/null' ) ) & } ; sleep 9 ; kill -INT $(pstree -p $! | sed -n 's/.*mongoexport(\([0-9]*\)).*/\1/p'); while wait -n ; do : ; done ) (s^13*t) xz -6 is 17.2978x time and .5913x size of gzip -6(s^7*t) xz -1 is 2.4042x time and .7802x size of gzip -6(s^1*t) gzip -2 is .5957x time and 1.1931x size of gzip -6

  • pstree dos comandos no slide anterior

    bashbashbashcat mongoexport tee10*[bashsed] timexz] wc] 9*[bashsed] timegzip] wc] bashbc tee3*[bashcut] grep] sort] tail]

  • cdigo bash, resultado do primeiro trecho

    mongoexport -d pseudo -c games 2>/dev/null | tee >(/usr/bin/time -f 't[11]=%U+%S' xz -1 | wc -c | sed 's/^/s[11]=/' 1>&2) >(/usr/bin/time -f 't[12]=%U+%S' xz -2 | wc -c | sed 's/^/s[12]=/' 1>&2) >(/usr/bin/time -f 't[13]=%U+%S' xz -3 | wc -c | sed 's/^/s[13]=/' 1>&2) >(/usr/bin/time -f 't[14]=%U+%S' xz -4 | wc -c | sed 's/^/s[14]=/' 1>&2) >(/usr/bin/time -f 't[15]=%U+%S' xz -5 | wc -c | sed 's/^/s[15]=/' 1>&2) >(/usr/bin/time -f 't[16]=%U+%S' xz -6 | wc -c | sed 's/^/s[16]=/' 1>&2) >(/usr/bin/time -f 't[17]=%U+%S' xz -7 | wc -c | sed 's/^/s[17]=/' 1>&2) >(/usr/bin/time -f 't[18]=%U+%S' xz -8 | wc -c | sed 's/^/s[18]=/' 1>&2) >(/usr/bin/time -f 't[19]=%U+%S' xz -9 | wc -c | sed 's/^/s[19]=/' 1>&2) >(/usr/bin/time -f 't[21]=%U+%S' gzip -1 | wc -c | sed 's/^/s[21]=/' 1>&2) >(/usr/bin/time -f 't[22]=%U+%S' gzip -2 | wc -c | sed 's/^/s[22]=/' 1>&2) >(/usr/bin/time -f 't[23]=%U+%S' gzip -3 | wc -c | sed 's/^/s[23]=/' 1>&2) >(/usr/bin/time -f 't[24]=%U+%S' gzip -4 | wc -c | sed 's/^/s[24]=/' 1>&2) >(/usr/bin/time -f 't[25]=%U+%S' gzip -5 | wc -c | sed 's/^/s[25]=/' 1>&2) >(/usr/bin/time -f 't[26]=%U+%S' gzip -6 | wc -c | sed 's/^/s[26]=/' 1>&2) >(/usr/bin/time -f 't[27]=%U+%S' gzip -7 | wc -c | sed 's/^/s[27]=/' 1>&2) >(/usr/bin/time -f 't[28]=%U+%S' gzip -8 | wc -c | sed 's/^/s[28]=/' 1>&2) >(/usr/bin/time -f 't[29]=%U+%S' gzip -9 | wc -c | sed 's/^/s[29]=/' 1>&2) | cat > /dev/null

  • t[29]=0.93+0.03s[29]=1725598t[28]=1.09+0.00s[28]=1728327t[27]=0.53+0.01s[27]=1779913t[26]=0.42+0.01s[26]=1817520t[25]=0.40+0.06s[25]=1868768t[24]=0.28+0.06s[24]=2001968t[23]=0.26+0.01s[23]=2068779t[22]=0.20+0.00s[22]=2167255t[21]=0.29+0.01s[21]=2257515

    scale=4;for(p=1;p

  • entrada do ultimo script bash

    1.7356 (s^1*t) xz -0 is 1.9565x time and .8871x size of gzip -61.7317 (s^1*t) xz -1 is 2.2173x time and .7810x size of gzip -62.9388 (s^1*t) xz -2 is 3.9565x time and .7428x size of gzip -63.9534 (s^1*t) xz -3 is 5.5000x time and .7188x size of gzip -65.6251 (s^1*t) xz -4 is 6.9782x time and .8061x size of gzip -67.9124 (s^1*t) xz -5 is 10.7608x time and .7353x size of gzip -611.5197 (s^1*t) xz -6 is 19.5217x time and .5901x size of gzip -611.5289 (s^1*t) xz -7 is 19.6304x time and .5873x size of gzip -611.6693 (s^1*t) xz -8 is 19.8695x time and .5873x size of gzip -611.7460 (s^1*t) xz -9 is 20.0000x time and .5873x size of gzip -6.6479 (s^1*t) gzip -1 is .5217x time and 1.2420x size of gzip -6.4924 (s^1*t) gzip -2 is .4130x time and 1.1923x size of gzip -6.7174 (s^1*t) gzip -3 is .6304x time and 1.1381x size of gzip -6.8379 (s^1*t) gzip -4 is .7608x time and 1.1014x size of gzip -61.0056 (s^1*t) gzip -5 is .9782x time and 1.0281x size of gzip -61.0000 (s^1*t) gzip -6 is 1.0000x time and 1.0000x size of gzip -61.4688 (s^1*t) gzip -7 is 1.5000x time and .9792x size of gzip -62.1495 (s^1*t) gzip -8 is 2.2608x time and .9508x size of gzip -62.5589 (s^1*t) gzip -9 is 2.6956x time and .9493x size of gzip -6.8457 (s^7*t) xz -0 is 1.9565x time and .8871x size of gzip -6.3929 (s^7*t) xz -1 is 2.2173x time and .7810x size of gzip -6.4933 (s^7*t) xz -2 is 3.9565x time and .7428x size of gzip -