Python packaging and dependency resolution

Python Packagingand how to improve dependency

resolution

Tatiana Al-Chueyr Martins@tati_alchueyr

PythonDay Pernambuco - 28th September 2013, Recife

tati_alchueyr.__doc__

● computer engineer by UNICAMP

● senior software engineer at Globo.com

● open source enthusiastic● pythonist since 2003

#opensource #python #android #arduino

Packaging overview

Code Package PackageServer

Code

# life.py

def cleanup_house(address): # ....

def walk_dog(dog_name): # …

class Man(object): __doc__ = “”

Package

# world.py

from life import Man

Package Server

https://pypi.python.org/

Packaging overview

Code Package PackageServer

Packaging creation

pack upload

Packaging creationpack

# setup.py

(...)

$ python setup.py sdist

Packaging creationupload

# ~/.pypirc[company]username: Andreiapassword: pwdrepository: http://pypi.company.com

$ python setup.py sdist upload -r company

Packaging usage

search &download

use

Packaging usagesearch &download

use

$ easy_install life

# something.py

from life import Man

Package

way to make the code available so developers can use it

Package

setup.py- contains lots of metadata- dependencies- paths

Packages server: Cheese Shop

place where developers can:● find packages● download packages● upload packages

Brief on Python packaging history● distutils

○ no dependecy management○ problems between cross-platforms○ no consistent way to reproduce an installation○ not all metadata was handled

● setuptools: built on top of distutils○ introduces easy_install○ no way to uninstall installed packages○ provides dependencies management○ introduced eggs (similar to zip files)

● distribute: fork of setuptools○ fork of setuptools

● distutils2 (discontinued?)○ standard versioning (major.minor.micro)○ setup.cfg: pulls metadata from setup.py file, without needing to run setup.py○ which operating system requires which dependecy

pysetup: their interations easyinstall and setuptools with disutils- extract stuff from setup.py

Distutils

● Started by Distutils SIG (Greg Ward)● Added to stand lib in Python 1.6 (2000)● solves

○ issues a variety of commands through setup.py (crete tarball, install your project, compiling C extensions of your python code)

● problems○ no dependency management○ problems between OS○ no consistent way to reproduce an installation○ not all metadata was handled

Brief on Python packaging history

PEP 386: changing the version comparison modules

PEP 376: database of installed python distributions

PEP 345: metadata for python software packages 1.2

Chronology of Packaging by Ziade

http://ziade.org/2012/11/17/chronology-of-packaging/

Chronology of Packaging by Ziade

http://ziade.org/2012/11/17/chronology-of-packaging/

Brief on Python packaging history

old busted new hawtnesssetuptools -> distributeeasy_install -> pipsystem python -> virtual-env

Virtualenv

“virtualenv is a tool to create isolated Python environments.”

https://pypi.python.org/pypi/virtualenv

VirtualenvWrapper

“virtualenvwrapper is a set of extensions to Ian Bicking's virtualenv tool” -- Doug Hellmann

https://pypi.python.org/pypi/virtualenvwrapper

$ mkvirtualenv <name>--python=--no-site-packages=--system-site-packages=

$ rmvirtualenv

$VIRTUALENVWRAPPER_HOOK_DIR/initialize

Pip

A tool for installing and managing Python packages.

http://www.pip-installer.org/en/latest/index.html

$ pip search numpy$ pip help$ pip install flask$ pip uninstall django$ pip freeze--no-deps--extra-index-url --index-url--download-cache --proxy --no-installgit / egg / ...pip install -r requirements.txt

Pip

(...)“This allows users to be in control of specifying an environment of packages that are known to work together.”(...)

http://www.pip-installer.org/en/latest/cookbook.html

How Pip deals with dependencyinconsistencies?

Pip install -r requirements.txt

B

A

# requirements.txtBC

# B/setup.pyA==1.0.0

# C/setup.pyA>=2.0.0 C

what version of A is installed?$ pip install -r requirements.txt

Pip install -r requirements.txt

# requirements.txtBC

# B/setup.pyA==1.0.0

# C/setup.pyA>=2.0.0

$ pip freezeA==1.0.0B==1.0.0C==1.00.

B

A

C

Pip install -r requirements.txt

# requirements.txtCB

# B/setup.pyA==1.0.0

# C/setup.pyA>=2.0.0

what happens? error?$ pip install -r requirements.txt

B

A

C

Pip install -r requirements.txt

# requirements.txtCB

# B/setup.pyA==1.0.0

# C/setup.pyA>=2.0.0

$ pip freezeA==2.0.0B==1.0.0C==1.00.

B

A

C

Pip install -r requirements.txt

# requirements.txtCBA==1.5.0

# B/setup.pyA==1.0.0

# C/setup.pyA>=2.0.0

what happens? error?$ pip install -r requirements.txt

B

A

C

Pip install -r requirements.txt

# requirements.txtCBA==1.5.0

# B/setup.pyA==1.0.0

# C/setup.pyA>=2.0.0

$ pip freezeA==1.5.0B==1.0.0C==1.00.

B

A

C

Explanation

Considering pip 1.5.4:● pip doesn’t identify conflicts of interest

between dependency packages● why?

○ pip solves dependencies analyzing them in a list○ it only concerns in solving the dependencies of the

package being analyzed at that moment○ the last package dependencies prevail

provided a package at pypi, how do I know its dependencies?

provided a package at pypi, how do I know its dependencies?

manually looking to them

dependencies of a package

if you install a package, you can use:$ pip show C

To show dependencies, but they don’t contain versions - only packages names

use pipdeptree$ pip freezeA==1.0.0B==1.0.0C==1.0.0

$ pipdeptree Warning!!! Possible confusing dependencies found:* B==1.0.0 -> A [required: ==1.0.0, installed: 1.0.0] C==1.0.0 -> A [required: >=2.0.0, installed: 1.0.0]------------------------------------------------------------------------wsgiref==0.1.2B==1.0.0 - A [required: ==1.0.0, installed: 1.0.0]C==1.0.0 - A [required: >=2.0.0, installed: 1.0.0]

Does the requirements.txt assure your environment will be reproduced always the same?

Does the requirements.txt assure your environment will be reproduced always the same?

not necessarily

requirements.txt

if you want to assert the same behavior in all installations:● don’t use >=, <=, >, <● pin all dependencies (even deps of deps)● pin exactly (==)

some extra notes

Have your own pypi / proxy

old versions might be removed from remote repositories

the repository might be down during a deploy, and can crash your application

Have your own pypi / proxy

Have your own pypi / proxyhost a PyPI mirror (bandersnatch, pep381client)host a PyPI cache (devp)

PyPI server implementations:● resilient (devpi)● AWS S3 PyPI server (pypicloud)● minimalistic PyPI (pypiserver)● PyPI written in Django (chishop, djangopypi)

Many others..!

At globo.com we have both a PyPI server and a PyPI cache proxy.

dumb ways to manage your dependencies….

1. understand the tools you use to manage dependencies2. keep your dependencies up to date, but take care with >= / >3. take care of your cheese-shop

use pipdeptree package!

thanks!

slideshare: @alchueyr

questions?

Tatiana Al-Chueyr Martins@tati_alchueyr

last notehttp://pypi-ranking.info/author