CIS192 Python Programming - Bash Script Replacement

  • View
    213

  • Download
    0

Embed Size (px)

Text of CIS192 Python Programming - Bash Script Replacement

  • CIS192 Python ProgrammingBash Script Replacement

    Eric Kutschera

    University of Pennsylvania

    April 17, 2015

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 1 / 27

  • Final Project

    Sign up for a demo slotLink is on the website for week 14Email me if none of the demo slots work for your group

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 2 / 27

  • Outline

    1 Bash Script ReplacementBashEmulating BashBetter than Bash

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 3 / 27

  • Why use Bash

    Youre probably already running your code from some sort ofterminal

    If youre not you should beMake full use of the terminal by learning bash commands

    Plug for CIS 191 (Linux Unix Skills)Easily chain together a bunch of programs

    ps -ef | grep -i python | wc -l

    Build up small programs with a Bash script .shHas features of a basic programming languageVariables, loops, conditionalsSave frequent command sequences to reuse later

    Example: Bash script to open connection to airPennNet

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 4 / 27

  • Why Bash is Not so good

    Scripts easily get overly complicatedSyntax is awkward:

    Update variable: lines=$(($lines + 5))iterate an array: for x in ${arr[@]}

    Hard to reuse code. (No imports)Minimal use of data typesIf you need to use control flow

    if [[ ! -z /file/path ]]Many others besides -z: -abcdefghkprstuwDifference between [[ ]] and [ ]

    In general very confusingExample: Package manager script I once understood

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 5 / 27

  • Outline

    1 Bash Script ReplacementBashEmulating BashBetter than Bash

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 6 / 27

  • Running from the command line

    A Bash script would be run from the command line./script.sh or sudo script.sh

    In order to figure out how to run the scriptThe #! (Shebang) line is readIt must be the first line of the fileIt says what interpreter to use#!/usr/bin/bash

    To execute a Python file with ./script.pyPut a shebang on the first line#!/usr/bin/python3

    Allow the current user to execute the filechmod u+x file.py

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 7 / 27

  • subprocess

    The subprocess module allows execution of shell commandssubprocess.call(ls)

    The commands are run in a child processLonger commands can be specified with a list of strings

    call([grep, -ir, python, ./])

    The I/O of the subprocesses can be set with kwargscall(ls, stdin=f_handle, stdout=DEVNULL)

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 8 / 27

  • Converting Bash Script

    A simple Bash script that is just a list of commandsAnd the Python equivalent

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 9 / 27

  • Alternatives to Call

    subprocess.callExecutes the commandWaits until it exitsReturns the exit code

    subprocess.check_callJust like call but ...If the exit code is not one (Abnormal exit)raise a CalledProcessError

    subprocess.check_outputJust like check_call but ...returns the contents of stdout after the process finishes

    subprocess.PopenTakes the same arguments as call (Except timeout)Doesnt wait for the process to finishHave to check for output explicitlyMore flexible for more complicated tasks

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 10 / 27

  • glob

    Shell globbing allows selection of files according to a limited regex* matches 0 or more of any character? matches 1 of any character[a-z] character ranges in brackets**/ matches any number of directories

    A convenient way to select filesEssentially walks the directory applying a modified regexfrom glob import glob, iglob

    glob(path_regex) returns a list of matching filesiglob(path_regex) an iterator version of glob

    Python glob doesnt support **/But it will in Python3.5Workaround is to use os.walk

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 11 / 27

  • tempfile

    Why Use a temporary fileMany interfaces require a file-like objectSome interfaces require an actual fileInter-process communication

    tempfile.TemporaryFile(rt)Creates a temporary file on the file systemGuaranteed not to conflict with existing filesSupports similar arguments to open(filename)Can specify a prefix and/or a suffix:

    preXXXXXsuffix where Xs are random numbers

    Deletes the file when it is closed or garbage collected

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 12 / 27

  • More tempfiles

    named_temp = tempfile.NamedTemporaryFile()Guaranteed to be a visible file on the file systemCan access its name with named_temp.name

    tempfile.SpooledTemporayFile(max_size=n)Wont actually write to disk unless the size exceeds max_size

    tempfile.TemporaryDirectory()A temporary directoryReturns a tuple f_handle, absolute_dir_pathWont automatically get deleted

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 13 / 27

  • Piping

    Using subprocess you can pipe the output of one processto the input of another processThe easiest way: Just use | in a subprocess command

    subprocess.check_call([ls, |, wc, -l])

    subprocess.PIPE lets you do this with more controlMust use subprocess.Popen instead of subprocess.call

    subprocess.call waits for the process to finishIf the pipe fills up then the process will deadlock

    from subprocess import Popen, PIPEp1 = Popen([ls], stdout=PIPE)p2 = Popen([wc, -l], stdin=p1.stdout,

    stdout=PIPE)p1.stdout.close()output = p2.communicate()[0]

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 14 / 27

  • Outline

    1 Bash Script ReplacementBashEmulating BashBetter than Bash

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 15 / 27

  • Command Line Arguments

    In Bashyou can access command line args with $1, $2, ...$0 is the name of the script

    In PythonAccess args with sys.argv[1], sys.argv[2], ...sys.argv[0] is still the name of the script

    What if you want to variable number or order of arguments?Bash has getopts which works but is still pretty messyPython has argparse

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 16 / 27

  • argparse

    from argparse import ArgumentParser

    argparse makes it easy to define command line argsAnd it will print a nice help message for youCreate a parser object with a description of the programs function

    parser = ArgumentParser(description=some text)

    Specify the arguments you expect to take inparser.add_argument(pos_arg_1)

    Parse the argsargs = parser.parse_args()print(args.pos_arg_1)

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 17 / 27

  • Positional and Keyword args

    parser.add_argument has lots of featuresThe first argument is the name of the argument to be parsedadd_argument(pos_arg)

    Creates a required positional argumentCan be accessed with args.pos_arg

    add_argument(-o, --optional)Creates an optional flag argumentPassed on the CL with -o value or --optional valueCan be accessed with args.optional

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 18 / 27

  • Argument Actions

    By default add_argument stores the CL value in argsThe action taken for each argument can be set withadd_argument(action=)

    action=store is the default behavioradd_argument(action=store_const, const=val)

    Stores val in argsMakes most sense for --flag arguments

    add_argument(action=store_true)Equivalently:add_argument(action=store_const, const=True)

    Also add_argument(action=store_false)

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 19 / 27

  • More Actions

    add_argument(action=append)Stores a list in argsIf CL is --f 1 --f 2 args.f == [1, 2]

    add_argument(action=append_const, const=val)Appends a constant value to a listIf CL is --f --f args.f == [val, val]Useful when also using the dest argument

    add_argument(action=count)Counts the number of times a flag is passedadd_argument(-v, --verbose, action=count)If CL is -vvv args.verbose == 3

    add_argument(action=version, version=1.0.0)Prints the version and then exits

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 20 / 27

  • nargs and default

    add_argument(nargs=)N (an integer): expects exactly N things and puts them in a list?: 0 or 1. Uses the value if given else uses a default*: 0 or more. If none are given, the default is used+: 1 or more. If none are given, an error message is givenargparse.REMAINDER: Put all unused args in a list

    add_argument(default=)Arguments omitted are given the default valueThe kwarg default is None if not setadd_argument(default=argparse.SUPPRESS)Causes nothing to be added to args for that flag

    Eric Kutschera (University of Pennsylvania) CIS 192 April 17, 2015 21 / 27

  • type, choices, and required

    add_argument(type=)Can specify any one argument function to be appliedadd_argument(--foo, type=int)If CL is --foo 12 args.foo == int(12)argparse.FileType allows pre-applying arguments to openadd_argument(type=FileType(w)) opens for writing

    add_argument(choices=)Restricts valid arguments to a setadd_argument(choices={2, 7, 29})Will print an error message if argument is not in choices

    add_argument(required=)Arguments of form -f, foo are not required by defaultMake them required with add_argument(required=