When working on any python project or data science project it is essential to always work in an environment that makes your project reusable and repeatable without any issues for anyone that picks up your project in the future.
This will even help you yourself when you revisit your project in the future. One of the ways to solve this issue is to use a virtual environment. The reason is that there are two main types of packages and locations where your python libraries resides and you do not need all of these packages when working on certain project hence it is required to know which one is required per project to make it easier for the reproducibility. These include
- System Packages that forms part of the Standard Python Library
- Site Packages (Third Party Packages) that your install using pip.
So what is a virtual environment?
A Virtual environment is an isolated workspace which keeps your packages separate from your local(main) system installation . It allows you to create a “virtual” isolated environment for each python projects. This makes it easier for each project to be independent on the other project especially where they share the same dependencies. There are various packages that can be use to create a virtual environment. Among them includes
Now after creating a virtual environment for your project how do you get the installed packages and libraries? With the virtual environment it is quite easy to get the precise packages you used in that project. Let us see how.
Working with Virtualenv
Virtualenv is a library that allows you to create a virtual environment. To install and work with it you can simply do
pip install virtualenv
Create a new working directory for your project
Create a new virtual environment inside the new project directory
python3 -m venv name_of_env
In order to use this isolated environment you will need to activate it using source
You will notice a change in your prompt with a prefix of whatever name you used for your virtual environment
To install any package or library for your current project you can use pip3 or pip
How to Get the Requirements.txt File: Using Virtualenv
Now to be able to get the requirements.txt file you can now use the pip freeze or pip3 freeze (python3) command as below
pip3 freeze > requirements.txt
How to Get the Requirements.txt File: Using Pipenv
Pipenv is also an awesome virtual environment creation library that has some cool features. To work with it you can use
# Install pip install pipenv # Install Your Packages for the project pipenv install mypackage # Activate Virtual Env pipenv shell # Run a script in the virtual env pipenv run python myscript.py
First of all it quite easier and it also automatically keep tracks of all the libraries used for the project in a Pipfile and a Pipfile.lock file. These files plays the same role as a requirements.txt plus some additional things.
Hence you can use it inplace of your requirements.txt file. But in case you want a requirements.txt file you can use
pipenv -r lock >> requirements.txt
How to Get the Requirements.txt File: Without VirtualEnv using Pipreqs
Pipreqs is the other simple alternative to use that doesn’t require you to create a virtual environment first. This is quite useful if you need only the packages and libraries used in an application or script you used.
It automatically scans the python file or script for their imported libraries and then generates a requirements.txt file. Let use see how to work with it
pip install pipreqs
To get the requirements.txt file you can then start pipreqs and point it to your folder with the python file
It is that simple right?
So what if you are working with conda or pipenv. Pipreqs makes it easier you can also use it like that.
Thanks you for your time