How does Python setuptools work?

However, the environment where the apps run would be dynamic. That means there are chance the automation process would be failing.

Once that happened, we tend to do troubleshooting. Troubleshooting would take more time if you don’t understand the basic of how stuff works. So, through this article I want to show how python setuptools work from different angle.

When I says different angle, that means from the point where setuptools normally deployed in the system. So whenever things not going well, you know where to start looking.

safe_lxml

I am going to use safe_lxml as simple application to install.

edx/edx-platform

You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…

github.com

The application is part of OpenEDX platform project. It simply provide safer version of lxml.etree by overrides some unsafe functions from lxml.etree with safer versions from defusedxml. It also includes a safer XMLParser.

Here’s the directory structure of safe_lxml.

You can read the full explanation of how to compose setup.py from this link, for the sake of simplicity, above content simply define some parameters for setuptools. I assume you already have setuptools installed on your system. So whenever you run something like this,

$ python setup.py install

The script will import setup from setuptools, read the parameters such as name of package to be created (safe_lxml), and what are the requirements to be installed. In this case, lxml and defusedxml must be installed.

As per description of safe_lxml above, the package provide safer version of lxml.etree — that is part of original lxml package, by overriding function with the one provided by defusedxml package. So that means both original package must be available on the system. OpenEDX project use safe_lxml instead of original lxml package.

You can create small project, compile them into binary, and distribute to your friend so he / she can call some function of your project to be used inside their own project.

Python setup tools perform many task automatically to facilitate distribution of your project so you don’t have to write down the step to your friend of how to compile the project since setuptools will do it automatically for your friend as long as you and your friend follow the standard defined by python setuptools. It will read the configuration in setup.py, prepare some directories, compile the script, create the binary, copy into correct directories, downloading dependencies, etc.

Let see what python setuptools do to install the safe_lxml package (I use MacOS).

Create Python Eggs

Python eggs is a single-file importable distribution format. It is pretty much similar to static-linked (binary) library in C language.

First, setuptools will create directory called safe_lxml.egg-info. Several files will be created inside that directory such as,

• PKG-INFO: contain information of the package such as its Metadata-Version, Name, Version, etc. As you might already guess, that information extracted from parameters define in setup.py file
• requires: contain information about name of dependency package for this python egg. Again, it is extracted from setup.py file
• SOURCES: contain information about all related files to build the package such as original python file, the directory listing information, etc

Next, setuptools will compile the app — all files under safe_lxml, to produce .pyc file. But before that, setuptools will prepare some directories first.

It will create directory build/lib/safe_lxml, and copy the app from safe_lxml directories into that new directory. Since my operating system is MacOS, setuptools will create suitable directories structure and files such as macosx-10.15 (Catalina) and the architecture is x86_64. As for the .pyc itself, setuptools will add -38, that means I use python version 3.8.

Kindly look at above output. I’ve structured them into (hopefully) more readable format so you can easily understand the context.

In the final step, dist package will be created in a zipped format containing all processed directories.

The dist package is an egg package that is ready to be distributed to anyone else. In this case, setuptools will install the egg package into my python environment (site_package) so safe_lxml can be used by other apps later.

Installing Dependencies

The last part is to install dependencies:defusedxml and lxml. Again, above output should be pretty clear, kindly have a look. It simply search the package from pypi and install into python site_package in local environment.

Again, by understanding how setuptools work especially the flow inside the operating system whenever a package deployed, you can have clearer minds about what might goes wrong whenever automatic installation of packages failing.

Setuptools give pretty rich log of information during the installation so it is always worth to read the lines especially in the case where you can’t find any answers from googling 😊