How I setup my python environment

Create robust and maintainable projects using the right tools

Photo by Chris Ried on Unsplash

After 9 years of Python development, I tested different ways to install Python and configure my projects. For those who are rather junior or even curious, to avoid you tearing your hair out on these issues as I did, I will present you the tools I have been using for a few years to manage my Python projects.

Install Python

Linux / Unix

If you don’t want any headaches, just use pyenv. It is a great tool for installing any version of CPython¹. It can also help to install alternative interpreters like PyPy or GraalPython. You can read my tutorial on it.

Pyenv: your Linux / Unix tool to manage different python versions

Windows

There is an adaptation of pyenv for Windows called pyenv-win. Honestly, I never used it. I just downloaded and installed the Python version I wanted on the Python website.

After that, if I have many versions of Python installed (3.7, 3.8, etc…) and I want to switch from one to another, I use the py command installed with the interpreter. Here is a nice introduction to this launcher.

Note: If you want to use binary packages or compiled extensions like numpy, you will need to have a C compiler that CPython can use. The simplest way I found to have a compatible one is to install Visual Studio (at least with C/C++ support). Yeah! it is ugly and it will take you some space on the hard disk, but it works 100 % of the time. By the way, if you have another solution that you are sure works on Windows, feel free to tell me in the comments.

Scaffold a Python project

Packaging and dependency management

The first tool we will need to start a Python project is poetry. It is a modern packaging and dependency management tool largely more productive to use than the traditional pip. You can look at the article I wrote on this package at the following link.

Note: poetry does not adhere to PEP 621. This is normal since it was born before this PEP and already has its style for declaring dependencies. If you prefer tools that are compatible with this PEP, you can choose between PDM and hatch. I never used any of them but from the little I’ve read on the two, PDM seems more complete and it has a lock file like poetry.

Lint

Here are the various tools you may want to use to simplify the readability of your code, and respect standards like PEP 8.

Black: The most popular autoformatting tool in the Python ecosystem.

Blue: Another autoformatting tool based on Black. I have a personal preference for this library since it defaults to single quotes. 🥹

You can read more about formatting tools in this article.

Black and Blues

Isort: A great tool to sort imports.

Flake8: A source code checker avoiding many pitfalls in coding like unused variables.

Ruff: The new shining library in the Python linting ecosystem. It does the job of Flake8 and isort and in the future, it will do the job of Black. I have an introduction to it here.

Python: lint code faster

Security

bandit: A great tool to detect some security flaws in your code like using the random module to create security tokens (it is better to use the secrets module for this purpose).

Ruff: Yeah, again! In fact, Ruff implemented all the rules of bandit.

safety: An awesome tool to detect packages with vulnerabilities in your project. Bear in mind that this library has two licenses. One allows for free use for open-source projects and another requires one to subscribe to a paid plan for commercial products.

pip-audit: Like safety, but the database used is open-source and the library is entirely open-source.

Here is a nice introduction by me to these scanning tools.

Auditing your python environment

Continuous Integration / Continuous Deployment (CI/CD)

pre-commit: If you need to do some actions before and after a commit, this tool comes in handy. Keep in mind to don’t abuse the tools you want to run in your pre-commit configuration file. If the tool you run in your pre-commit file can be easily checked in your CI/CD environment, it is better to check it there to not slow your productivity.

nox: A great piece of software not so much known in my humble opinion. It is a successor to tox allowing us to run our project and tests on a matrix of different Python interpreters. I have a tutorial on it here.

Nox: the shining python test automation tool

Tests

pytest: The first and obvious package for testing. It is more powerful than the built-in unittest framework and has a large ecosystem of plugins.

pytest-cov: The pytest plugin to check the coverage of your project. If you are using the debugger of your editor with this plugin, bear in mind that it will not probably work well. You will need to disable pytest-cov when testing with your favorite debugger.

Documentation

A good project needs excellent documentation. Here are the tools I can recommend to you to achieve this goal.

mkdocs: A great package for documentation using the Markdown syntax.

mkdocstring: A mkdocs plugin to create documentation from docstring in your source code.

mkdocs-material: Another mkdocs plugin to create a responsive documentation website for your project. Recently, it adds the possibility to host a blog post within the site.

sphinx: Another great tool for documentation leveraging the reStructuredText syntax. It can also work with Markdown but I never used it, so I won’t say more about it.

Bonus

I don't want to be pretentious to tell you how to code but if there is a library that I advise you to use in every project it is Pydantic. It is a data validation and settings management library using Python type annotations. At the moment of writing, the team maintaining this library is working on the V2 where the core is written in Rust, promising better performance. In the following blog posts, I present some less-known features of Pydantic.

Brief history of dataclasses in Python

Hidden power or Pydantic

If you want to develop a command-line application, I recommend you the click library. It is well-written and really simple to use. Here are other tutorials I wrote on this topic. 🙃

Click: a beautiful python library to write CLI applications

Click-params: a little companion to your click project

Also, for your web development projects, there are plenty of choices possible but the two that I prefer are:

FastAPI: A web framework leveraging type annotations. If you prefer a lightweight yet powerful solution it is my recommendation. Combined with ORMs like SQLAlchemy or Prisma, you can quickly write robust applications.

FastAPI and pagination

Alchemical, the unknown friend of SQLAlchemy

Alternatives to SQLAlchemy for your project — Prisma case

Django: Probably the most complete in the Python ecosystem. It is my way to go for all my serious projects 🙃. The learning curve is a bit high compared to other frameworks, but when you have the basics, you are really productive.

For REST² JSON APIs, you can combine it with Django Rest Framework + DRF-Spectacular or Django Ninja + Extras to have a FastAPI approach.

Summary

In this article, I have summarized 9 years of trials, tests, and continuous improvement to:

• Install Python

• Manage dependencies

• Write clean code

• Check vulnerabilities in packages and code source

• Test the source code

• Document the project

• And some tips on nice packages I use daily.

If you want a concrete example where I use most of the tools listed above, you can check the source code of my websocket project.

This is all for this article, hope you enjoy reading it. Take care of yourself and see you soon. 😁

1. CPython is the main Python interpreter, but there are alternatives like PyPy

2. There has been a deviation in the meaning of REST, this article explains what it really means.