CIS192 Python Programming - Bash Script Replacement

CIS192 Python ProgrammingBash Script Replacement

Eric Kutschera

University of Pennsylvania

April 17, 2015

Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 1 / 27

Final Project

Sign up for a demo slotLink is on the website for week 14Email me if none of the demo slots work for your group


Outline

1 Bash Script ReplacementBashEmulating BashBetter than Bash


Why use Bash

You’re probably already running your code from some sort ofterminal

If you’re not you should beMake full use of the terminal by learning bash commands

Plug for CIS 191 (Linux Unix Skills)Easily chain together a bunch of programs

ps -ef | grep -i python | wc -l

Build up small programs with a Bash script .shHas features of a basic programming languageVariables, loops, conditionalsSave frequent command sequences to reuse later

Example: Bash script to open connection to airPennNet


Why Bash is Not so good

Scripts easily get overly complicatedSyntax is awkward:

Update variable: lines=$(($lines + 5))iterate an array: for x in ${arr[@]}

Hard to reuse code. (No imports)Minimal use of data typesIf you need to use control flow

if [[ ! -z /file/path ]]Many others besides -z: -abcdefghkprstuwDifference between [[ ]] and [ ]

In general very confusingExample: Package manager script I once understood


Outline



Running from the command line

A Bash script would be run from the command line./script.sh or sudo script.sh

In order to figure out how to run the scriptThe #! (Shebang) line is readIt must be the first line of the fileIt says what interpreter to use#!/usr/bin/bash

To execute a Python file with ./script.pyPut a shebang on the first line#!/usr/bin/python3

Allow the current user to execute the filechmod u+x file.py


subprocess

The subprocess module allows execution of shell commandssubprocess.call(’ls’)

The commands are run in a child processLonger commands can be specified with a list of strings

call([’grep’, ’-ir’, ’python’, ’./’])

The I/O of the subprocesses can be set with kwargscall(’ls’, stdin=f_handle, stdout=DEVNULL)


Converting Bash Script

A simple Bash script that is just a list of commandsAnd the Python equivalent


Alternatives to Call

subprocess.callExecutes the commandWaits until it exitsReturns the exit code

subprocess.check_callJust like call but ...If the exit code is not one (Abnormal exit)raise a CalledProcessError

subprocess.check_outputJust like check_call but ...returns the contents of stdout after the process finishes

subprocess.PopenTakes the same arguments as call (Except timeout)Doesn’t wait for the process to finishHave to check for output explicitlyMore flexible for more complicated tasks


glob

Shell globbing allows selection of files according to a limited regex* matches 0 or more of any character? matches 1 of any character[a-z] character ranges in brackets**/ matches any number of directories

A convenient way to select filesEssentially walks the directory applying a modified regexfrom glob import glob, iglob

glob(path_regex) returns a list of matching filesiglob(path_regex) an iterator version of glob

Python glob doesn’t support **/But it will in Python3.5Workaround is to use os.walk


tempfile

Why Use a temporary fileMany interfaces require a file-like objectSome interfaces require an actual fileInter-process communication

tempfile.TemporaryFile(’rt’)Creates a temporary file on the file systemGuaranteed not to conflict with existing filesSupports similar arguments to open(’filename’)Can specify a prefix and/or a suffix:

’preXXXXXsuffix’ where X’s are random numbers

Deletes the file when it is closed or garbage collected


More tempfiles

named_temp = tempfile.NamedTemporaryFile()Guaranteed to be a visible file on the file systemCan access its name with named_temp.name

tempfile.SpooledTemporayFile(max_size=n)Won’t actually write to disk unless the size exceeds max_size

tempfile.TemporaryDirectory()A temporary directoryReturns a tuple f_handle, absolute_dir_pathWon’t automatically get deleted


Piping

Using subprocess you can pipe the output of one processto the input of another processThe easiest way: Just use ’|’ in a subprocess command

subprocess.check_call([’ls’, ’|’, ’wc’, ’-l’])

subprocess.PIPE let’s you do this with more controlMust use subprocess.Popen instead of subprocess.call

subprocess.call waits for the process to finishIf the pipe fills up then the process will deadlock

from subprocess import Popen, PIPEp1 = Popen([’ls’], stdout=PIPE)p2 = Popen([’wc’, ’-l’], stdin=p1.stdout,

stdout=PIPE)p1.stdout.close()output = p2.communicate()[0]


Outline



Command Line Arguments

In Bashyou can access command line args with $1, $2, ...$0 is the name of the script

In PythonAccess args with sys.argv[1], sys.argv[2], ...sys.argv[0] is still the name of the script

What if you want to variable number or order of arguments?Bash has getopts which works but is still pretty messyPython has argparse


argparse

from argparse import ArgumentParser

argparse makes it easy to define command line argsAnd it will print a nice help message for youCreate a parser object with a description of the program’s function

parser = ArgumentParser(description=’some text’)

Specify the arguments you expect to take inparser.add_argument(’pos_arg_1’)

Parse the argsargs = parser.parse_args()print(args.pos_arg_1)


Positional and Keyword args

parser.add_argument has lots of featuresThe first argument is the name of the argument to be parsedadd_argument(’pos_arg’)

Creates a required positional argumentCan be accessed with args.pos_arg

add_argument(’-o’, ’--optional’)Creates an optional flag argumentPassed on the CL with -o value or --optional valueCan be accessed with args.optional


Argument Actions

By default add_argument stores the CL value in args

The action taken for each argument can be set withadd_argument(action=)

action=’store’ is the default behavioradd_argument(action=’store_const’, const=val)

Stores val in argsMakes most sense for ’--flag’ arguments

add_argument(action=’store_true’)Equivalently:add_argument(action=’store_const’, const=True)

Also add_argument(action=’store_false’)


More Actions

add_argument(action=’append’)Stores a list in argsIf CL is --f 1 --f 2 → args.f == [1, 2]

add_argument(action=’append_const’, const=val)Appends a constant value to a listIf CL is --f --f → args.f == [val, val]Useful when also using the dest argument

add_argument(action=’count’)Counts the number of times a flag is passedadd_argument(’-v’, ’--verbose’, action=’count’)If CL is -vvv → args.verbose == 3

add_argument(action=’version’, version=’1.0.0’)Prints the version and then exits


nargs and default

add_argument(nargs=)N (an integer): expects exactly N things and puts them in a list’?’: 0 or 1. Uses the value if given else uses a default’*’: 0 or more. If none are given, the default is used’+’: 1 or more. If none are given, an error message is givenargparse.REMAINDER: Put all unused args in a list

add_argument(default=)Arguments omitted are given the default valueThe kwarg default is None if not setadd_argument(default=argparse.SUPPRESS)Causes nothing to be added to args for that flag


type, choices, and required

add_argument(type=)Can specify any one argument function to be appliedadd_argument(’--foo’, type=int)If CL is --foo 12 → args.foo == int(’12’)argparse.FileType allows pre-applying arguments to openadd_argument(type=FileType(’w’)) opens for writing

add_argument(choices=)Restricts valid arguments to a setadd_argument(choices={2, 7, 29})Will print an error message if argument is not in choices

add_argument(required=)Arguments of form ’-f’, ’–foo’ are not required by defaultMake them required with add_argument(required=True)


dest and metavar

add_argument(dest=)Specify the attribute name to be used in argsBy default uses the positional name or longest option flagUseful for storing multiple arguments to the same dest

add_argument(metavar=)metavar is used when printing the help messageBy default, argparse will use the dest value as the name


CSV files

Comma Separated Value (CSV) files are a simple data formatMost programs can export a CSV file (Excel, databases)Most programming languages can read/write CSV filesIn general a safe way to share datafrom csv import DictReader

list_of_dicts = list(DictReader(open(’csvfile’)))

from csv import DictWriter

writer = DictWriter(open(’dest’, ’w’), headers)writer.writeheader()for d in dicts:

writer.writerow(dict)


Excel Spreadsheets

Pandas can read and write Excel filesUses xlrd and xlwt for old Excel files (.xls)Uses openpyxl for Excel 2010 files (.xlsx)Optionally uses XlsxWriter if installed for xlsx (better charts)

Or you can use the Excel libraries directlyA little bit cumbersomeNecessary if you want to use Excel specific features

FormulasFont sizes, color


Sending Emails

A common thing to do in Bash scripts is send email notificationsIn Bash mail or mailx can be used

mail [email protected] -s ’subject’<<< ’message’Have to setup some config filesCouldn’t get it to work on my machineWorks on Eniac and will send mail from your pennkey@seas


Python Email

Use the standard libraries smtplib and email

Can send email through a Gmail account with object interface

import smtplibfrom email.mime.text import MIMEText

server = smtplib.SMTP(’smtp.gmail.com:587’)...msg = MIMEText(’contents of message’)...server.send_message(msg)server.quit()


Documents

CIS192 Python Programming - Bash Script Replacement