Python 3.5: An agile, general-purpose development language

NAP TalksPython 3.5: An agile, general-purpose development language.

Speaker: Carlos Miguel FerreiraPosition: PhD Student and IT ResearcherContact: carlosmf.pt@gmail.com

Layout1st Part – The Framework

➔ Author➔ Philosophy➔ The Interpreter➔ The Global Interpreter Lock➔ Python Enhancement Proposals (PEPs)➔ Eggs Management

BreakTime

2 49/

2nd Part – The Basics➔ Built-in Types➔ Built-In Functions➔ Modules➔ Methods➔ Objects➔ Exceptions➔ Lambdas➔ Operators

3rd Part – Doing Stuff in Python➔ Networking➔ Serializing Data➔ Persisting Data➔ File Access➔ Threads➔ Python Interpreter Arguments➔ Assertions➔ Byte Compiling Python Code➔ String Operations

1st Part

The Python Development Framework

3 49/

Python HistoryThe Author:➔ Guido Van Rossum➔ Currently, the Python’s Benevolent Dictator

Objective➔ A descendant of ABC that would appeal to Unix/C hackers.

1st Implementation (CPython)➔ Version 0.9➔ Started in December, 1989➔ During the Christmas vacations, Rossum was bored...

Why Python?➔ He’s a Fan of Monty Python's Flying Circus.

Other Python Implementations➔ JPython (Java)➔ Iron Python (C#)➔ PYMite (Micro-controllers)

4 49/

Guido Van Rossum at Open Source Convention in 2006 (OSCon 2006)

Python PhilosophyThe Zen of Python, by Tim Peters

Beautiful is better than ugly.Explicit is better than implicit.Simple is better than complex.Complex is better than complicated.Flat is better than nested.Sparse is better than dense.Readability counts.Special cases aren't special enough to break the rules.Although practicality beats purity.Errors should never pass silently.Unless explicitly silenced.In the face of ambiguity, refuse the temptation to guess.There should be one-- and preferably only one --obvious way to do it.Although that way may not be obvious at first unless you're Dutch.Now is better than never.Although never is often better than *right* now.If the implementation is hard to explain, it's a bad idea.If the implementation is easy to explain, it may be a good idea.Namespaces are one honking great idea -- let's do more of those!

5 49/

Go ahead!Open a Python console and type import this

Tim at Python Convention in 2006 (PyCon 2006)

The Python Interpreter – How it Works 6 49/

/bin/python3 helloWorld.py

Python compiles the code into byte instructions

python35.dll libpython3.5.so

The code

The Python interpreter runs the byte code

libpython3.5.dylib Python is linked with the respective system libs, providing access to the system resources.

Console Output

Python does not use a Virtual Machine like Java. There is no Machine Abstraction.

The Python language establishes a programming interface, common to different operative systems.

Python’s Global Interpreter Lock 7 49/

In order to keep Python’s Interpreter implementation simple, some issues exist:➔ The memory management is not thread-safe➔ This means, no synchronization is made for shared-

memory regions.

Global Interpreter Lock➔ It serializes the thread access to the interpreter.➔ Only one thread may use the interpreter at any

time.➔ Because of this, Python Threads cannot use

multiple cores at the same time.➔ This issue is related only to CPython

Other Python Interpreters do not suffer from GIL➔ JPython (Java implementation)➔ Iron Python (C# implementation)

Thread 1

Thread 2

Thread 3

Thread N

GIL Synchronizes Thread’s Access to Interpreter.

GIL

➔ Python Threads are in fact System Native Threads.

➔ Python just creates an abstraction.➔ It is the OS that schedules the threads to

execute, not Python.

Python Enhancement Specification - PEP

For more information: PEP-1 and PEPs

8 49/

➔ The Python Enhancement Specification can be:● Standards Track Basically a standard→● Informational Describes issues, provides guidelines or informations→● Proposal Proposals for new additions, pending reviews. →

➔ From time to time, the community proposes changes or new additions to the Python Language/Framework.

➔ Each new proposal is fulfilled by creating a new PEP.➔ A PEP can be either accepted or rejected.

There are many PEP’s!➔ PEP 1 PEP Purpose and Guidelines→➔ PEP 8 Style Guide for Python Code→➔ PEP 20 The Zen of Python→➔ PEP 101 Doing Python Releases 101→

The Python Package Index - The Eggs Keeper

For more information: Setup Tools and PyPI

9 49/

Currently, there are thousands of packages, available to Python.

These packages can be packaged into EggsThere are two types of Eggs➔ Pure Only contain Python code (→ cross-platform).➔ Hybrid Contain both Python code and binary shared objects (→ specific

to target architecture).

Examples:➔ PySerial Pure Module for Serial Port Management→➔ NumPy Hybrid Module. Contains C-Optimized Mathematical Structures →

and Algorithms implementations.

Python Eggs can be:➔ Created using the python setuptools and distributed as a single

container.➔ Distributed with the developed program itself.

Who is using Python?

For more information: Cases of Success

10 49/

Python is used around the World!"Python is fast enough for our site and allows us to produce maintainable features in record times, with a minimum of developers," by Cuong Do, Software Architect, YouTube.com.

"Python has been an important part of Google since the beginning, and remains so as the system grows and evolves. Today dozens of Google engineers use Python, and we're looking for more people with skills in this language." by Peter Norvig, director of search quality at Google, Inc.

2nd Part

The Basics11 49/

Variables and Objects 12 49/

Variable Object

Python Is a Dynamic typing Language➔ Variables do not have a defined type at compile time.➔ Variables can reference any type of object!➔ Type checking is performed at run-time!

● Because of this, dynamic languages are slower than static languages.

Variable Object Value

Type

Python➔ Object is instantiated and variable X gets

the reference.➔ The variable type is dynamic in the sense

that it can change in run-time.

C➔ Variable type is known at compile time

and does not change during run-time.➔ It is statically defined.

Python Objects 13 49/

Everything in Python is an Object!

➔ object is the base type for every Python Object.

➔ Every object has an identity● In CPython3.5, it’s object memory address.● Obtained by using the id(obj) built-in.

➔ Object type can be obtained with type(obj) built-in.

➔ Objects can be:● Mutable: value can be changed.● Immutable: value is unchangeable.

➔ Object mutability is defined by its type.

Objects Are Never Explicitly Destroyed!➔ Allocation is done by the interpreter.

● Same goes with deletion➔ Object existence in memory is managed

using reference counter.➔ When reference counter reaches zero,

objects are deleted.

Beware with reference cycles!➔ Python can optionally detect cyclic

references but it is not guaranteed that it may work.

For more information: Data Model

Mutable and Immutable Objects 14 49/

For more information: Data Model and Built-In Objects

Remember → Object mutability is defined by its type.

Number (immutable)➔ Integral

➔ Integers → x = 123➔ Booleans → x = True or x = False

➔ Real → x = 999.666➔ Complex → x = (-1+0j)

Sequences➔ Immutable sequences

➔ Strings → x = “Hello World!!!”➔ Tuples → x = (123 , 999.999 , “Hello!”)➔ Bytes → x = b“1234567”

➔ Mutable sequences➔ Lists → x = [123 , 999.999 , “Hello!”]➔ Byte Arrays → x = bytearray()

Set Types (sets theory)➔ Object references do not repeat inside set.➔ Objects must be immutable.➔ Sets (mutable)

➔ x = set((1 , “2” , 3.3))➔ Frozen Sets (immutable)

➔ x = frozenset((1 , “2” , 3.3))

Mappings➔ Dictionaries (mutable)

➔ x = {key : data}➔ Keys must be hashable → hash(obj)➔ x = {1:2, "list":[1,2]}

There are other types so check them.

Hello World! 15 49/

Result

Importing module sys➔ It provides the .argv property, which is a list,

containing the program arguments.

Importing Python Modules 16 49/

For more information: List of the Python Interpreter Core Modules

Python is structured in Modules.From these modules, it is possible to extend the python interpreter capabilities.

There are several ways to import modules:By directly import➔ import time

By specifically importing its functions➔ from time import sleep, time, ctime

By importing everything (namespace polution)➔ from time import *

Modules present in the default installation

sys → System-specific parameters and functions.io → Core tools for working with streams.os → Miscellaneous operating system interfaces.threading → Higher-level threading interface.multiprocess → Process-based “threading” interface.socket → Low-level networking interface. asyncio → Asynchronous I/O, event loop, coroutines and tasks.cmd → Support for line-oriented command interpreters.datetime → Basic date and time types.time → Time access and conversions.heapq → Heap queue algorithm.queue → A synchronized queue class.pathlib → Object-oriented filesystem pathsmath → Mathematical functionssqlite3 → DB-API interface for SQLite3 Databases.

and more...

Python Built-in Functions 17 49/

For more information: Core Functions

Built-in functions are core functions, provided by the Python Interpreter.

These functions are implemented in C and because of that, are capable of using and manipulating Python objects at memory level, with increased performance.

The interpreter provides “binds” to access these core functions.

Python Reserved Keywords 18 49/

For more information: Keywords

Lets take a look at some of the reserved keywords

True, False ➔ Unique Boolean Objects

None➔ The “nothing” object (evaluates to False)

if, elif, else➔ Logical Statements

and, or➔ Logical operators (equivalent to &&, || )

class, def ➔ To define classes, functions and methods

respectively

for, while➔ Loops

continue, break➔ For loops execution control

try, except, finally➔ For exceptions usage and control

in➔ Reference search in container/iterator.

del➔ To delete variable from scope➔ To execute __del__ method from objects.

User Defined Functions → def 19 49/

Result

Method with required argument

Method with optional argument

Method with (by order):1) Required argument2) Optional argument3) Extra unnamed arguments4)Extra named arguments

Order of arguments declaration, must be obeyed.

For more information: Defined Function

See this empty space? These are actually tabs!Python uses tabs ( |) for code indentation!→

Scopes → nonlocal, global 20 49/

For more information: Scopes and Namespaces

Result

➔ A scope is a textual region of a Python program where a namespace is directly accessible.

➔ A namespace is a mapping from names to objects.Name = “String Object”

Variables have names which relate to objects

Namespaces keep the relationships.

At least three scopes are directly accessible1. The innermost scope, which is searched first, contains

the local names.2. The scopes of any enclosing functions, which are

searched starting with the nearest enclosing scope, contains non-local, but also non-global names.

3. The next-to-last scope contains the current module’s global names.

4. The outermost scope (searched last) is the namespace containing built-in names (built-in functions).

if Statement 21 49/

Result

For more information: If Statement

Chaining conditional evaluations with if, elif, else.

Checking the type of an object using the built-in type().

Comparing object identifications using the built-in id().

for and while Loops 22 49/

ResultResult

For more information: For Loops and While Loops

Exceptions → try, except 23 49/

Result

For more information: Errors and Exceptions

Objects & Classes 24 49/

Every object in python, inherits from objectobject is the everything and the nothing (None)

Python Objects can have:➔ A generalized constructor.➔ Instance Attributes.➔ Class Attributes.➔ Instance Methods.➔ Class Methods.➔ Operators.

Classes are defined using the class statement.➔ By default, classes inherit from object➔ Classes in Python also support multiple

inheritance.

For more information: Classes, Private Variables, Multiple Inheritance

Object Construction → class 25 49/

For more information: Classes, Private Variables and Emulating Numeric Types

Result

Object generic constructor

Object Method

Class Method Has Class object as →parameter

Static Method

Called when del(obj)

About Object Casting

There’s no such thing as Casting in Python!

Calling built-in functions such as int(), float(), str() with an object as argument, will respectively call instead, the methods __int__(self), __float__(self) and __str__(self), implemented by the object.

class statement

Objects Properties 26 49/

For more information: PropertiesResult

Running the code...

Two different ways to do the same thing

Objects Operators 27 49/

For more information: Basic Customization and Operators

Result

> operator

< operator

Conversion to string C→ alled when str(obj)

Used to obtain an object representation Called when repr(obj) or when __str__ does not exist.

Polymorphism The right way→ 28 49/

For more information: Inheritance

Module that provides abstraction tools

abstraction class configuration

Abstract methods are required to be implemented. If not...Result

Generators 29 49/

Python Generators are simple but powerful expressions to create iterators.

Result

For more information: Generators

➔ Python Generators implement the __iter__() and __next__() methods automatically.

➔ Request a value from a generator by calling next() or by using a for-loop to iterate through the values.

➔ yield statement stops the generator execution and returns a value.

➔ Execution resumes after yield statement.➔ When return is called, the generator

terminates.➔ Terminated generators raise StopIteration.

Fibonacci Numbers Generators

yield stops the generator execution and returns a number

return terminates generator

Built-in range iterator

Generator Expressions 30 49/

Python Generator Expressions are one-line version of generators.

Result

Generator Expression in one line

Can be used in practical situations, like reading lines from files

For more information: Generator Expressions

Lambdas 31 49/

Python Lambdas are anonymous, one-lined simplified functions.

Lambda with (x, y, z, length) as arguments.When a lambda is created, it is kept in a function object.Result

For more information: Lambda

Lambda with x as argument and created in-line.

3rd Part

Doing Stuff in Python32 49/

File Access 33 49/

File Access in Python is Trivial

Result

Result

Reading lines, one by one using iteration.

Reading a File Writing a File

Reading entire file.

File modes:➔ “r” Read→➔ “w” Read and Write→➔ “a” Append at end if exist, if not, →

create file.➔ “b” Binary Mode→

With statement can be used to create an inner context, closing the file automatically, upon closure.

For more information: Reading and Writing Files, IO and with statement

Threads 34 49/

Python Threads are native OS threads.

For more information: Threads

Result

Synchronization problem.

Thread main function

Creating a Thread objecttarget function to execute→args function arguments →

Waiting for Threads to end

Since this is a simple example, the args could have been declared in the global namespace and accessed using the global statement.

I did this on purpose, since it was my intention to use the target arguments.

Synchronizing Threads with Locks 35 49/

Locks can be used for synchronization.

For more information: Locks

Correct ResultLock object

with statement using Lock

The Lock object provides an acquire() and release() functions.

For this example, the with statement was used on purpose to present a cleaner and simple code.

Persisting Data with pickle 36 49/

Sometimes, it is necessary to save data...

For more information: Pickle

dumps serializes the object, returning a bytestring

loads, given a bytestring, de-serializes the object.

It is simple to serialize a Python object

It is also simple to persist an object into a file.dump serializes the object into a file.➔ File object is provided by the open() built-inload, given a file object, loads from the object.

Result

Binary data with struct 37 49/

For networks, every byte counts...

For more information: Struct and Pickle Format

unpack reverses the process➔ The same format rules apply.➔ Unpack always returns a tuple, even if it only has 1

element.

pickle can be used to create a message, to send over the network:➔ 2 bytes for the Length of the Data field (“H”).➔ “s” indicates a sequence of bytes, preceded by

the length.

pack creates a bytestring with a specific format➔ “I” specifies an unsigned integer with 4 bytes.➔ “!” specifies the byte order should be in Big Endian.➔ Not specifying the byte-order, will default to system

endianness.

assert Statement 38 49/

For more information: Assert Statement

Assertions are a good way to prevent code misuse.

Result

➔ The assert statement takes a logical evaluation argument and a string.

➔ The Built-in isinstance() is used to determine if arg is a string instantiated object.

➔ If the assert logical evaluation fails, an AssertionError exception is raised, containing the assertion argument string.

Efficient String Manipulation 39 49/

For more information: Strings

Result

Formatting Decimal values

Formatting Float

Adjusting value alignment

Joining strings

Custom strings

Formatting Decimal values➔ # Alternate version→

Efficient Dictionary Manipulation 40 49/

For more information: Dictionary

Result

Sorting

Python Interpreter Arguments 41 49/

For more information: Command Line and Docstring PEP

Arguments can be passed to the Python Interpreter

Useful arguments:

➔ -m Search for the named module and execute →its contents as the main __main__ module.

➔ -O Turn on basic optimizations. Changes the →filename extension for compiled (bytecode) files from .pyc to .pyo.

➔ -OO Discard → docstrings in addition to the -O optimizations. Docstrings are special strings embedded in methods or

modules, which provide information about their use.

Result

Calling Help

Byte Compiling Python Code 42 49/

For more information: Compile All

Python code can be byte-compiled priory to execution.

Calling the compieall module, to byte compile the program.py file.

Directly executing the byte compiled file.

Working with the Network → socket 43 49/

For more information: Socket

Python provides binds to lower-level OS sockets.

Lets start with sending packets through a Layer 2 Socket.

Result

Executing

Sudo is needed due to the required extra permissions

Working with the Network → socket 44 49/

For more information: Socket

Reading all packets from a Layer 2 Socket.

Result

Receiving up to 1514 bytes (Max MTU).

Unpacking MAC Address and Ether Type.

Working with the Network → socket 45 49/

For more information: Socket

Now, lets send UDP Packets to the Network.

Result

Working with the Network → socket 46 49/

For more information: Socket Server

ResultResult

Client – Server interaction.

Client Side Server Side

Python is (almost) everywhere! 47 49/

AstroPy

Books and Webdocs of Interest!Books:➔ Automate the Boring Stuff with Python: Practical Programming for Total Beginners➔ Python for Raspberry Pi (2014)➔ Beginning Programming with Python For Dummies (2014)➔ Foundations of Python Network Programming, 3rd Edition, (Apress, 2014)➔ High Performance Python: Practical Performant Programming for Humans (O’Reilly, 2014)➔ Python for Finance: Analyze Big Financial Data - (OReilly, 2014)➔ Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython

WebDocs:➔ Python Official Documentation➔ SciPy➔ MatplotLib➔ SimPy➔ PyBrain➔ PyGame➔ PyOpenGL

48 49/

We Are Done!

Thank you for your attention!49 49/