Pip and Virtual Environments with venv in Python

Note: If you don’t know what Pip is and how to use it, read this article first: Pip: Python’s package-management system.

To do serious programming in Python you need to use Virtual Environments.

When developing new projects, eventually you have to install and manage dependencies with pip.

But if you just do pip install requests, for instance, you will install it globally on your computer.

This means that every project you use has access to that dependency in that version, which is the latest release in this case since we didn’t specify a version.

But what if one dependency crashes with another?

What if your Project X uses requests version 2.23.0 and some old Project Y uses requests 1.0.1?

If you don’t have a virtual environment, this might become a real problem.

It is a standard to use a dedicated virtual environment for each Python project.

A virtual environment will create an isolated environment for each project, so each one has its own dependencies.

You will see most projects in Python out there use the setup we are going to learn in this article, so you will be able to recognize this pattern as soon as you see a file named ‘requirements.txt’.

The tool we are going to use is called venv, it comes installed by default with your Python 3 installation.

Create a directory ‘my_project’ and enter inside it.

mkdir my_project
cd my_project

Then create a new virtual environment using venv.

python3 -m venv my_venv

Notice I called it the folder where our virtual environment will live ‘my_venv’.

Your folder structure shoud be similar to this.

+-- my_project
|   +-- my_venv
|   |   +-- bin
|   |   +-- include
|   |   +-- lib
|   |   +-- lib64
|   |   +-- pyvenv.cfg

The structure inside ‘my_venv’ is created automatically.

The lib folder is where the independent packages will live along with the Python versions itself.

The bin folder has the tools to start and stop the virtual environment.

To activate the virtual environment, execute:

source my_venv/bin/activate

Your terminal will be displayed with a prefix on the beginning that matches the name of the virtual environment which is my_venv if you used the same name as I did.

My terminal looks like this:

(my_venv) renan@pro-home:~/my_project

To stop the virtual environment and go back to your regular system terminal, call deactivate.

deactivate

The prefix (my_venv) will disappear.

Activate it again with source my_venv/bin/activate.

Let’s try some of the pip commands we already know inside the virtual environment.

If we type pip list to display the currently installed packages, it will look like this.

Package    Version
---------- -------
pip        19.2.3 
setuptools 41.2.0 
WARNING: You are using pip version 19.2.3, however version 20.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Notice that we have only pip and setuptools installed, and there is even a warning to upgrade pip.

Upgrade pip with python -m pip install --upgrade pip.

If you run pip list now, there will be no more warnings.

Try to install Beautiful Soup inside the virtual environment with pip install beautifulsoup4 and pip list the packages once again.

Notice how compact our list of packages is, with only what we really need.

Package        Version
-------------- -------
beautifulsoup4 4.9.1
pip            20.1.1
setuptools     41.2.0
soupsieve      2.0.1

After doing a lot of work in your virtual environment, installing packages and all, you want to transport it somewhere else.

How do you guarantee that someone else or even yourself will have the same packages installed, with the same versions, to run your project properly?

pip has a command called freeze, this command that outputs the installed packages in your terminal, so you just need to redirect this output using > to a file.

pip freeze > requirements.txt

You can call the file by any name, but ‘requirements.txt’ is a convention, that is why I said on the beginning of this section that you will recognize this pattern of pip and venv as soon as you see a ‘requirements.txt’ file in a project.

If you look at the contents of ‘requirements.txt’ and followed exactly what I did, you should see the following:

beautifulsoup4==4.9.1
soupsieve==2.0.1

The versions might differ, but you should see these packages in the list.

This means that you can easily create a brand new and clean virtual environment somewhere else and use pip install using -r to indicate the reference file containing all the packages you want to install.

To try it out, deactivate the virtual environment and create a new folder outside ‘my_project’ called ‘new_project’.

mkdir new_project
cd new_project

Copy the file ‘requirements.txt’ from ‘my_project’ to ‘new_project’.

Create a virtual environment inside ‘new_project’ with python3 -m venv venv.

Activate it with source venv/bin/activate.

With the virtual environment activated, run pip install -r requirements.txt, and then pip list.

Package        Version
-------------- -------
beautifulsoup4 4.9.1  
pip            19.2.3 
setuptools     41.2.0 
soupsieve      2.0.1 

Now you have a new environment with the same packages of ‘my_project’ in ‘new_project’.

You can also uninstall them all at once with

pip uninstall -r requirements.txt -y

The -y is simply to always confirm when it asks "Are you sure you want to uninstall package ?".

You can also specify that you want newer versions of the packages if available, just change from = to >=.

In our case, we could change ‘requirements.txt’ to:

beautifulsoup4>=4.9.1
soupsieve>=2.0.1

This means beautifulsoup4 will be installed with at least version 4.9.1.

To force an upgrade after adjusting the ‘requirements.txt’ with:

pip install --upgrade -r requirements.txt

So if there is a beautifulsoup4 version superior to 4.9.1, this newer version will replace the current one.

Finally, you can control which packages you want to install in a production environment and which ones in a development environment.

The idea is to create a second file ‘requirements-dev.txt’.

Again, this is a convention, you could call it by any name.

Inside ‘requirements-dev.txt’ you specify that you want to install everything in ‘requirements.txt’ with -r requirements.txt and some additional packages you need only for development.

A common situation for this is when you use the pytest library, which is used to test your code and thus is only needed for development, not a production environment.

The ‘requirements-dev.txt’ file will have the following content.

-r requirements.txt
pytest>=5.4.3

So if you are in a development environment, you run pip install -r requirements-dev.txt and it will install everything inside ‘requirements.txt’ and the extra packages, in this case, only pytest.

If you are in a production environment, you run pip install -r requirements.txt, and the extra packages in ‘requirements-dev.txt’ won’t be installed.

There is an alternative to venv called pipenv, you can learn about it in this article: Virtual Environments in Python with Pipenv

This article is a sample chapter from the book Python Fundamentals.