Don't be a GUI rookie forever - Be in command! (line)

Don’t be a rookie forever.Be in command! (line)

Giuseppe Maxia

@datacharmerThis work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.

$Sunday, August 21, 11

about me -Giuseppe Maxia§ a.k.a. The Data Charmer§ QA Director, Continuent, Inc§ Long time hacking with database features§ Formerly, database consultant, designer, coder.§ A passion for QA§ An even greater passion for open source§ ... and community§ Passionate blogger§ http://datacharmer.blogspot.com

Sunday, August 21, 11

http://datacharmer.blogspot.com




















What this tutorial is about§ Providing tools and tips for command line users;§ Learn a few tricks to make you achieve more and type

less.


What this tutorial is about§ Providing tools and tips for command line users;§ Learn a few tricks to make you achieve more and type

less.§ Making GUI enthusiasts green with envy;

bar

foo

baz


This tutorial§ A quick guide at the command line usage§ A practical case for database admins§ but ...§ Useful for all


Problem§ One document contains one date in 'mm/dd/yyyy'

format§ Need to convert the date into the 'yyyy-mm-dd' format

(or dd.mm.yyyy)

GUI USER SOLUTION§ open the document with a word processor§ change the date

§ Command line USER SOLUTION§ Probably the same


Problem§ One document contains many dates in 'mm/dd/yyyy'

format§ Need to convert the dates into the 'yyyy-mm-dd'

format (or dd.mm.yyyy)

GUI USER SOLUTION§ open the document with an advanced word processor§ hope it supports regular expressions

§ Command line USER SOLUTION§ Use vi or emacs. Do a search-and-replace operation§ Or use the next page solution


Problem§ Many documents contain many dates in 'mm/dd/yyyy'

format§ Need to convert the dates in all documents into the

'yyyy-mm-dd' format (or dd.mm.yyyy)

GUI USER SOLUTION§ Realizes that it will take days to do this manually§ Scream

§ Command Line USER SOLUTION§ uses sed or perl to do the replacement in one line


the command line

WHY8

?


An educational story1985-901991199319941995-971998-992000-20012001-20062007-20092010-

9



9

• MS-DOS



9

• MS-DOS• Windows 3.0



9

• MS-DOS• Windows 3.0• Windows 3.1



9

• MS-DOS• Windows 3.0• Windows 3.1• Windows NT



9

• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95



9

• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98



9

• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98• Windows 2000



9

• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98• Windows 2000• Windows XP



9

• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98• Windows 2000• Windows XP• Windows Vista



9

• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98• Windows 2000• Windows XP• Windows Vista• Windows 7



9

• MS-DOS• Windows 3.0• Windows 3.1• Windows NT• Windows 95• Windows 98• Windows 2000• Windows XP• Windows Vista• Windows 7

• Unix shell

• Linux

• Mac OSX


Why the command line• portability• efficiency• fun

10


the command line (for rookies)

NO PANIC

11

||||


Command line panic (for rookies)

$_Sunday, August 21, 11

Command line panic (for rookies)

$_NOW WHAT?


the command line (for rookies)

BASICS

13


The shell• It's a command interpreter• There are many flavors

> bash > zsh> ksh

14


MySQL command line client• called "mysql"

> Takes SQL commands> sends them to the server> gets results

• but it can do much more ...

15


other tools• Shell scripts• Aliases• Perl• awk• sort• …. and a lot of Unix little helper

16


other tools• Shell scripts• Aliases• Perl• awk• sort• …. and a lot of Unix little helper

16

which could replace everything else, of course


in case it wasn't clear (yeah, for rookies)

17



• command line actions are

17



• command line actions are> commands (the name of the application to invoke)

17



• command line actions are> commands (the name of the application to invoke)> parameters (optional arguments for the application)

17



• command line actions are> commands (the name of the application to invoke)> parameters (optional arguments for the application)> and the ENTER key

17



• command line actions are> commands (the name of the application to invoke)> parameters (optional arguments for the application)> and the ENTER key

17

Enter


The Unix philosophy (1)• One tool for every feature

> make one tool that does one thing well> for many features, make many tools> make them work together (see next slide)

18


The Unix philosophy (2)• To make several tool cooperate:

> pipe the output of one tool to the input of another one.> for more actions, pipe more tools

$ sort < filename | uniq | nl

19


Unix pipes

20


Unix pipes

20

inputapplication <


Unix pipes

20

outputapplication >

inputapplication <


Unix pipes

20

outputapplication >

inputapplication <

outputapplication >input<


Unix pipes

20

outputapplication >

inputapplication <


application | application


Unix pipes

20

outputapplication >

inputapplication <



input output


Unix pipes = parallel processing

21



21




21




21


output



21


input output



21


input output



21


input output output



21


input output output input


Who gets what• shell• application• database server• pager

22


who gets what

23

shell prompt $ mysql

mysql client

mysql server


any hostwho gets what

24

shellmysql client

mysql server


who gets what$ mysql -u username -ppassword

25

First the applicationthen the server


who gets what$ mysql -u username -ppassword \ -e "select VERSION()"

26

goes to the server

goes to the client, then the server


who gets whatmysql> helpmysql> pager lessmysql> select 1;

27

goes to the server

goes to the client onlygoes to the client only


the command line

readline28


readline• Common component

> shell> mysql client> many CLI applications

29


Main cursor movements

30

Ctrl A Ctrl E

Start/End of line

Meta B Meta F

Back/Forward one word

previous/next command

Ctrl L clear screen and take current line on top

Ctrl R Search history


Changing text

31

Meta U Uppercase word

Meta L Lowercase word

Ctrl D delete word

Ctrl K delete to end of line

Ctrl - undo

for more tips:$ man readline


Some conventions in this talk

$ commandat the shell

mysql> command at the mysql prompt

32


The command line friends

handytools33

#!


your best friends at the command line• man• ls• cd• find• grep• date• echo• pwd• mv• cp 34


man {command_name}• Tells you how to use a command

man ls

35


ls [directory]• lists contents of current [or given] directory

ls

ls /some/path

ls *.txt

36


cd [directory]• change directory to given path• takes to $HOME directory if no path is given

cd /some/path

cd ..

37


find {directory} [conditions]• finds files in the given directory that satisfy conditions

find $HOME/docs -name "*mouse*"

38


grep {pattern} filename• finds pattern matching rows in filename

grep Munich *.txt

grep "Munich\|Berlin" *.txt

grep "[0-9][0-9]:[0-9][0-9]" a.txt

39


date [layout]• displays the current date and time, with optional layout

dateTue Aug 16 08:33:04 CEST 2011

date '+%Y-%m-%d'2011-08-16

40


echo "message"• displays the given message and/or variables

echo "hi"hi

GREETINGS='Hello, there!'echo $GREETINGSHello, there!

41


pwd• displays the current path (Print Working Directory

pwd/home/gmax/test_dir

42


mv old_file_name new_file_name• moves a file from one place to another• renames a file

mv ugly.txt /some/other/pathmv *.txt /storage/texts

mv ugly.txt beautiful.txt

43


cp file_name destination• copies a file from one place to another• creates a copy of a file

cp ugly.txt /some/other/pathcp *.txt /storage/texts

cp ugly.txt beautiful.txt

44


other important friends at the command line

• rm• mkdir• rmdir• cat• sed• strings• wc• head• tail• more/less• sort

45


rm file_name• removes a file

rm ugly.txtrm *.txt

### DANGER!!!!rm -r * rm -rf *

46


mkdir/rmdir directory• creates or removes a directory

mkdir mydir

rmdir mydir

47


cat• shows the contents of a file

cat addresses.txt

48


sed [option] file • performs editing on a file

sed -e 's/Ship/Airplane/' travel.txt

49


strings filename• extracts printable text from a file

strings /bin/ls

50


wc filename• counts words (and characters and lines) in a file

wc .bashrc 72 164 2842 .bashrc# (lines, words, characters)

51


head/tail filename• display the first or the last lines of a file

head long_file.txthead -n 5 long_file.txt

tail long_file.txttail -n 1 long_file.txt

52


more/less filename• display a file with scrollable pages

# old: inefficientmore filename.txt

# new: preferredless filename.txt

53


sort filename• displays the ordered contents of a filename

sort unordered.txt

sort unordered.txt > sorted.txt

54


mixing languages

55

#!


This is a story about languages


you got to understand, Bernie, it's not personal.

It's just that you are obsolete

We use language to talk to computers


10 print "hello"20 goto 10We use

languages to talk to computers


I am a polyglot


An imperfect polyglot, actually


I speak Italian


I speak Italian

with a clear Sardinian accent


I speak English


I speak English

with a distinct Italian accent


I speak Perl


I speak Perl

with a hint of a C accent


I speak French and Spanish


I speak French and Spanishbut since I use them very little, they have melted with Italian and

EnglishSunday, August 21, 11

I speak Bash #!/bin/bash


I speak Bash #!/bin/bash

with a strong Perl accent


I speak fluent Italian

La donzelletta vien dallacampagna, in sul calar delsole, col suo fascio dell'erbae reca in mano un mazzolino di rose e viole, onde, siccome suole, ornare ella si appresta ...


However, Sardinian comes handy sometimes

Maissavidati frimisti! (*)

(*) May you never stopSunday, August 21, 11

I speak fluent English

When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.


http://en.wikipedia.org/wiki/Natural_law




hmm.wrong flag









But sometimes Italian is still necessary

maestro

virtuoso

al fresco

presto

spaghettibravo

tempo

dilettantefiasco

studio

stucco

barista

cappuccino

espresso

lasagnapizza

pasta

stanza

finale

lottostiletto

vendetta soprano piano


And sometimes Italian is needed

I tuoi perversidefunti!(*)

(*) Your wicked ancestors!Sunday, August 21, 11

Although Sardinian is still effective here

Ancu ti currullidisa giustizia (*)

(*) May the rule of law chase youSunday, August 21, 11

I speak Perl fluently

#!/usr/bin/perluse strict;use warnings;use File::Find;

while (my $line = <stdin>){ # …


Sometimes, I need help from C

use Inline C;print "9 + 16 = ", add(9, 16), "\n";

__END____C__int add(int x, int y) { return x + y;}


Most of my work at the command line is Bash

$ for D in $HOME/*/; do du -sh $D;> done > /tmp/occupancy

#!/bin/bash


And sometimes, a pinch of Perl can help

$ perl -pe 's{(\d\d)-(\d\d)-(\d\d\d\d)}{$3-$1-$2}g' \ < rawfile.txt > betterfile.txt

#!/bin/bash


And then on top of all this, I need to speak SQL

$ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" '


I wish I could say that my SQL has a Perlish accent


use strict;use DBI;?


I wish I could say that my SQL has a Perlish accent


but it wouldn't be true


I could curse in Perl

$ perl -pe 's{(\d\d)-(\d\d)-(\d\d\d\d)}{$3-$1-$2}g' \ < rawfile.txt > betterfile.txt (*)

(*) already translated

But it would not get me far.


Then


Then

I will need to mix languages.


Then


This talk will show exactly that.


Then



Mixing languages.


Then



Mixing languages.Yes.


Then



Mixing languages.

$ at the command line

Yes.


the Bash shell

bash81

#!


components• commands• aliases• variables• functions

82


commands$ echo "something"something

83


aliases$ alias mydate='mysql -e "select NOW()"'$ mydate+---------------------+| now() |+---------------------+| 2010-04-12 07:14:19 |+---------------------+

84


aliases$ alias myself='mysql' $ myself -e "select NOW()"'+---------------------+| now() |+---------------------+| 2010-04-12 07:14:19 |+---------------------+

85


variables$ MYBATCH='mysql -N -B'$ MYQUERY='select curdate()' $ $MYBATCH -e "$MYQUERY"2010-04-12

86


nesting variables# t.sh CMD="SELECT TABLE_NAME" TBL='information_schema.tables' WHC1="TABLE_SCHEMA='mysql'" WHC2="TABLE_NAME like 'u%' " Q1="$CMD FROM $TBL" Q2="WHERE $WHC1 AND $WHC2" Q="$Q1 $Q2" echo $Q echo $Q | mysql

87


nesting variables$ sh t.shSELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='mysql' AND TABLE_NAME like 'u%'TABLE_NAMEuser

88


the mysql command line client

89

mysql>


SQL commands from a file$ mysql < filename.sqlmysql> source filename.sql

• executes the commands contained in filename.sql

• commands can be queries understood by the server

• and commands for the client

90


SQL commands at the command line$ application | mysql$ echo "SELECT 1" | mysql$ mysql -e "SELECT 1"

• executes the commands from standard input

• commands can be queries understood by the server

• and commands for the client

91


batch mode$ echo "select curdate()" | mysqlcurdate()2010-04-12

92


table mode$ echo "select curdate()" | mysql -t+------------+| curdate() |+------------+| 2010-04-12 |+------------+

93


table mode$ mysql -e "select curdate()"+------------+| curdate() |+------------+| 2010-04-12 |+------------+

94


Forcing batch modemysql -B -e 'select curdate()'curdate()2010-04-12

95


Forcing batch mode - no headermysql -N -B -e 'select curdate()'2010-04-12

96

IMPORTANT!


Shell + mysql$ echo "The current server's date is `mysql -N -B -e 'select curdate()'`."

$ echo "The current server's date is $(mysql -N -B -e 'select curdate()')."

97


advanced command line techniques

loops98

$_

(-:((((((

@@


loops basics$ for N in 1 2 3> do> echo $N> done123

99


loops basics$ for N in 1 2 3 ; do echo $N ; done123

100


loops basics$ for N in $(seq 1 3) ;\ do echo "this is number $N" ;\ done

this is number 1this is number 2this is number 3

101


loops from file$ head /usr/share/dict/words AaaaaalaaliiaamAaniaardvarkaardwolf

102


loops from file$ for U in \$(head /usr/share/dict/words ) ; do perl -le 'print uc shift' $U ; doneAAAAAALAALIIAAMAANI

103


loops from mysql results$ mysql -B -N -e 'show tables from world'CityCountryCountryLanguage

104


loops from mysql results$ for T in $(mysql -B -N -e 'show tables from world') ;\> do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done

CREATE TABLE world2.City LIKE world.City;CREATE TABLE world2.Country LIKE world.Country;CREATE TABLE world2.CountryLanguage LIKE world.CountryLanguage;

105


piping loops back into mysql$ for T in $(mysql -B -N -e 'show tables from world') ;\> do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done \> | mysql -vv

106



nested

107

$_

@ @

#!

:-)

#!#!#!#!#!

loopsSunday, August 21, 11

doing something for every tablein every database$ mysql -e "show databases"+--------------------+| Database |+--------------------+| information_schema || mysql || sakila || world |+--------------------+

108


doing something for every tablein every database$ mysql -e "show tables from world"+-----------------+| Tables_in_world |+-----------------+| City || Country || CountryLanguage |+-----------------+

109


pseudocodefor each database do for each table do print the table name

110


actual codealias mybatch='mysql -B -N'for D in $(mybatch -e "SHOW SCHEMAS")do for T in \ $(mybatch -e "SHOW TABLES FROM $D") do echo "$D.$T" donedone

111



options files

112

$HOME


Default option files/etc/my.cnf/etc/mysql/my.cnf (Debian, Ubuntu)$BASEDIR/my.cnf$HOME/.my.cnf

113


Custom option files$ mysql --no-defaults$ mysql --defaults-file=filename$ mysql --defaults-extra-file=filename

114


Option files sections[mysqld]# option for server

[client]# option for any mysql client

[mysql]# options specific to 'mysql' CLI app

115


Option files example

[client]user=mynamepassword=mypassword

[mysql]prompt="mysql (\h) {\u} [\d] > "

116


Option files: caution# protect your option files!

$ chmod 600 $HOME/.my.cnf

$ ls -l .my.cnf-rw------- 1 [...] 10:04 .my.cnf

117


promptmysql> prompt gm (\h) {\u} [\d] > PROMPT set to 'gm (\h) {\u} [\d] > 'gm (localhost) {root} [(none)] > use testDatabase changedgm (localhost) {root} [test] >

# \h -> host# \u -> user# \d -> database

118



aliases119

#!

(-:

((((((@@


An useful alias$ alias mysql_as_root=\'mysql --defaults_file=~/.my_as_root.cnf'

mysql_as_root -e "select user()"+----------------+| user() |+----------------+| root@localhost |+----------------+

120


An useful alias#.my_as_root.cnf'[client]user=rootpassword=your_mysql_root_password

[mysql]prompt="### [\h] {\u} (\d) > "

121


Another useful aliasalias mybatch='mysql -B -N '

mybatch -e "select version()"5.1.45-log

122



functions123

#!

(-:

((((((@@( )


functions• like aliases, but with parameters

124


sample functionfunction grant_new { mysql_as_root -e \ "GRANT $1 on $2 to $3 identified by \"$4\" "}

$ grant_new SELECT "world.*" john \ john_secret

125


sample function$ mysql_as_root \ -e 'show grants for john\G'*** 1. row ***Grants for john@%: GRANT USAGE ON *.* TO 'john'@'%' IDENTIFIED BY PASSWORD '*73A88CE13374128844E3ABB500F439A9C02C53A3'*** 2. row ***Grants for john@%: GRANT SELECT ON `world`.* TO 'john'@'%'

126



getting query

details127

$_


which query is being executed?# wrong.sql# this script has one erroruse test ;drop table if exists t1 ;create table t1 (id int);insert into t1 values (a);

128


which query is being executed?$ mysql < wrong.sqlERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list'

# WHERE IS THE ERROR?

129


which query is being executed?$ mysql -v < wrong.sql--------------drop table if exists t1----------------------------create table t1 (id int)----------------------------insert into t1 values (a)--------------ERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list'

130


getting execution times$ mysql -vv -e \'select rating,count(*) from sakila.film group by rating'+--------+----------+| rating | count(*) |+--------+----------+| G | 178 || PG | 194 || PG-13 | 223 || R | 195 || NC-17 | 210 |+--------+----------+5 rows in set (0.00 sec) 131


$echo 'select rating,count(*) from sakila.film group by rating' | mysql -vv--------------select rating,count(*) from sakila.film group by rating--------------

rating count(*)G 178PG 194PG-13 223R 195NC-17 2105 rows in set

132

getting execution times (batch mode)


getting execution times (batch mode)$ echo 'select rating,count(*) from sakila.film group by rating' | mysql -vvv --------------select rating,count(*) from sakila.film group by rating--------------+--------+----------+| rating | count(*) |+--------+----------+| G | 178 || PG | 194 || PG-13 | 223 || R | 195 || NC-17 | 210 |+--------+----------+5 rows in set (0.00 sec) 133



getting distinct results

134

$_


getting a single value$ mysql -e "show variables like 'port'"+---------------+-------+| Variable_name | Value |+---------------+-------+| port | 3306 |+---------------+-------+

# Here, we only want the port number

135


getting a single value$ mysql -NB -e "show variables like 'port'" port 3306

# no header, batch mode# still two values

136


getting a single value$ mysql -NB \ -e "show variables like 'port'" \ | awk '{print $2}'

3306

$ mysql -NB \ -e "show variables like 'port'" \ | perl -lane 'print $F[1]'

3306 # filtering the wanted value

137


getting a single value: is replication working?

$ mysql \ -e "show slave status \G" \ | grep -i slave_io_running

Slave_IO_Running: Yes

# too many things. # We only want a 'yes' or 'no'

138


getting a single value: is replication working?

$ mysql \ -e "show slave status \G" \ | grep -i "slave_io_running" \ | perl -lane 'print $F[1]'

Yes

# yes!

139


What's perl -lane?

140


What's perl -lane?

140

perl -lane must turn perl


Perl command line must-know facts (1)perl -e 'print "hello\n"'# executes perl code

141


Perl command line must-know facts (2)perl -ne 'print if /â/'# starts a loop from standard input# and executes the perl code for each# iteration

perl -MO=Deparse -ne 'print if /â/'LINE: while (defined($_ = <ARGV>)) { print $_ if /â/;}

142


Perl command line must-know facts (3)perl -pe 's/(â\w+)/$1/'# like -ne, but prints each line

$ perl -MO=Deparse -pe 's/(â\w+)/$1/'LINE: while (defined($_ = <ARGV>)) { s/(â\w+)/$1/;}continue { print $_;}

143


Perl command line must-know facts (4)perl -le 'print "hello"'# strips newlines from input and adds # them to output

$ perl -e 'print "hello"'hello$ $ perl -le 'print "hello"'hello$ 144


Perl command line must-know facts (5)perl -lane 'print $F[0]'# splits input into @F array

$ echo "a b c" | \ perl -lane 'print $F[1]'b $ echo "a b c" | \ perl -lane 'print $F[2]'c 145


Perl and shell contention for $

146

Take the $$$ and run

Perl command line must-know facts (6)

#!/bin/bash


Beware of $VARIABLES on the command line$ perl -le "$x=1; print $x" syntax error at -e line 1, near "="Execution of -e aborted due to compilation errors.

147



147

consumed by the shell, before Perl can see it

double quotes allow the shell to interpolate variables



147







147


consumed by the shell, before Perl can see it$ perl -le '$x=1; print $x'

1





147


consumed by the shell, before Perl can see it$ perl -le '$x=1; print $x'

1


single quotes prevent variable consumption by the shell





combining column names

148


combining column names$ mysql -N -e 'desc world.City'+-------------+----------+----+-----+| ID | int(11) | NO | PRI || Name | char(35) | NO | | | CountryCode | char(3) | NO | | | District | char(20) | NO | | | Population | int(11) | NO | | +-------------+----------+----+-----+

149


combining column names# Expected result:# SELECT CONCAT_WS('|', # ID,Name,CountryCode,District, # Population) from City

150


combining column names (I)echo "select concat_ws('|', $(mysql -N -e 'desc world.City' | \ perl -lane 'push @f, $F[0];' \ -e 'END{print join ",", @f}')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Population) from world.City

151


combining column names (II)echo "select concat_ws('|', $( mysql -NB -e 'select group_concat(column_name) from information_schema.columns where table_name="City" and table_schema="world"')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Population) from world.City

152


combining column names (IIa)echo "select concat_ws('|', $( mysql -NB -e 'select group_concat(column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) from world.Country"select concat_ws('|', Code,Name,Continent,Region,SurfaceArea,IndepYear,Population,LifeExpectancy,GNP,GNPOld,LocalName,GovernmentForm,HeadOfState,Capital,Code2) from world.Country

153


piping the result back to mysqlecho "select concat_ws('|', $( mysql -NB -e 'select group_concat(column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) as line from world.Country" | mysql | head -n 4lineAFG|Afghanistan|Asia|Southern and Central Asia|652090.00|1919|22720000|45.9|5976.00|Afganistan/Afqanestan|Islamic Emirate|Mohammad Omar|1|AFNLD|Netherlands|Europe|Western Europe|41526.00|1581|15864000|78.3|371362.00|360478.00|Nederland|Constitutional Monarchy|Beatrix|5|NLANT|Netherlands Antilles|North America|Caribbean|800.00|217000|74.7|1941.00|Nederlandse Antillen|Nonmetropolitan Territory of The Netherlands|Beatrix|33|AN

154



dumping data

155

$_



dumping data

155

$_


dumping a tab delimited data set # the naive way$ mysql \ -e "SELECT * into outfile '/tmp/city_data' from world.City"

# it works only if:# - /tmp/city_data does not exist# - /tmp is world writable

156


dumping a tab delimited data set $ mysql -B -N \ -e "SELECT * from world.City" \ > /tmp/city_data

# it works always

# if you omit the "-N" it also # includes a header

157


dumping a CSV $ mysql -B \ -e "SELECT * from world.City" \ | perl -F"\t" -lane \ 'print join ",", map {s/"/""/g; /^[\d.]+$/ ? $_ : qq("$_")} @F ' \ > /tmp/city_data

158


dumping a CSV $ mysql -B \ -e "SELECT * from world.City" \ | perl -F"\t" -lane \ 'print join ",", map {s/"/""/g; /^[\d.]+$/ ? $_ : qq("$_")} @F ' \ > /tmp/city_data

158

splits the line by tabs, instead of the default whitespace



scriptit

159


use the mysql.sh script§ The companion code for this tutorial§ http://tinyurl.com/cli-tut-code-2010

§ It will save you much typing§ But use it at your own risk!

160


http://tinyurl.com/cli-tut-code-2010

http://tinyurl.com/cli-tut-code-2010

Usage• download the script• load it

161

$ source mysql.sh


mysql.sh functions• my_batch "query"• returns the result of the query. • Without header • Without box

162

$ my_batch "SELECT * FROM test.t1"1 aaa2 bbb3 ccc


mysql.sh functions• my_value "what"• returns a single value

• e.g.

163

$ my_value "version()"5.1.49

$ my_value "@@server_id"1


mysql.sh functions• my_var "what"• returns the value of a given variable

• e.g.

164

$ my_var port3306

$ my_var datadir/usr/local/mysql/data


mysql.sh functions• my_status "what"• returns the value of a given status item

• e.g.

165

$ my_status com_select2653

$ my_status connections452


mysql.sh functions• my_dump_dataset "query" [header]• returns the values of a given query separated by tabs• if a header is requested, it will be part of the result• e.g.

166

$ my_dump_dataset "select * from world.City" | head -n 51 Kabul AFG Kabol 17800002 Qandahar AFG Qandahar 2375003 Herat AFG Herat 1868004 Mazar-e-Sharif AFG Balkh 1278005 Amsterdam NLD Noord-Holland 731200


mysql.sh functions• my_csv "query" [separator]• returns the values of a given query separated by

commas, or by a given separator• e.g.

167

$ my_csv 'select * from world.City' |head -n 3"ID","Name","CountryCode","District","Population"1,"Kabul","AFG","Kabol",17800002,"Qandahar","AFG","Qandahar",237500


inside the command line client

handling

output168

$_

@ @

:-)


pagermysql> SELECT * FROM world.City;

rec …rec …rec …

4000 + recordsunreadable

169


pagermysql> pager lessPAGER set to 'less'mysql> SELECT * FROM world.City;

# [ records in a manageable window ]

mysql> nopagerPAGER set to stdout

# back to normal170


More pager tricksmysql> pager vim -# like 'less', but you can also edit# the results and eventually save # them

171


More pager tricksmysql> pager md5sumPAGER set to 'md5sum'mysql> select * from City;5d17ffa50d6da200dee621823ade2543 -4079 rows in set (0.01 sec)

# produce a CRC for the result set# instead of displaying it.# Useful for data comparison.# (Thanks to Baron Schwartz

172


Comparing data sets with md5summysql> select * from City;5d17ffa50d6da200dee621823ade2543 -

mysql> select * from City order by ID;5d17ffa50d6da200dee621823ade2543 -

mysql> select * from City order by name;5ab980f5b69a6e65d3b8cd5a95ec94e0 -

mysql> select * from City order by District;2bf37cbe91b3ffeaebc8c4b6084b8784 -

173


Comparing data sets with md5summysql> select * from City;5d17ffa50d6da200dee621823ade2543 -

mysql> select * from City order by ID;5d17ffa50d6da200dee621823ade2543 -

mysql> select * from City order by name;5ab980f5b69a6e65d3b8cd5a95ec94e0 -

mysql> select * from City order by District;2bf37cbe91b3ffeaebc8c4b6084b8784 -

173


More pager tricksmysql> pager cat > /tmp/t1.txtmysql> select "one" union select "two";mysql> pager cat > /tmp/t2.txtmysql> select "one" union select "TWO";mysql> nopagermysql> \! vimdiff -o /tmp/t[12].txt

# see the difference between two # datasets

174


175


logging sessionsmysql> tee mylog.txt

# logs everything that you see on# the screen.# your commands, the results,# warnings, error messages, and so on.

mysql> notee

# back to normal176


read more§ http://datacharmer.blogspot.com




THANKS!Question

timeComments on Twitter: @datacharmer

?

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.


to do§ add dead languages

§ comparing Latin and ancient Greek to COBOL and Pascal

§ add new tricks§ {x,y} expansion in bash§ \G in MySQL§


Technology

Don't be a GUI rookie forever - Be in command! (line)