Mirroring websites refer to the concept of replicating or copying a website or any network node with the purpose of
- Reducing network traffic, improve access speed
- Ensuring availability of the original site for technical or political reasons
- Providing a real-time backup of the original site.
In this tutorial we will be making a mirror of PyPI – the platform to fetch and download python packages. This is useful in case you want to have a private copy or internal use of PyPI. We will be using a package called DevPi.
Let us start
Setting Up Devpi-Server
To begin we will create a virtual environment and install devpi-server
pip install devpi-server
Once it is installed we will have to initialize devpi-server for it to create the required settings
Starting the Devpi Server
You can also use devpi-server –start to start the server. Once you do this you will see the port and the localhost open which the devpi-server is running. You can see the mirror_url that is being mirrored.
Devpi-server requires a configuration for running the server, copying the original files,etc. You can generate these configurations using the devpi-gen-config command
devpi-gen-config
Since it is a server, you will need supervisor to manage the start and stop of the server periodically. Hence you need to install supervisor via
sudo apt-get install supervisor supervisord -c gen-config/supervisord.confdevpi-gen-config
Installing Packages from the Devpi-Server
To install a package you can use
pip install -i http://localhost:3141/root/pypi/+simple/ thepackage_name
Now you have seen how to setup a local PyPI mirror server using DevPi.
To avoid re-typing the index URLs with pip when installing as above , you can configure pip by adding to the index-url entry in the pip.ini or pip.conf file. This is going to be global and be the one used when you use pip install.
The file can be found in your $HOME/.pip/pip.conf (unix) or $HOME/pip/pip.ini (windows). Then place in those file these
[global] index-url = http://localhost:3141/root/pypi/+simple/ [search] index = http://localhost:3141/root/pypi/
You can also add a special environment variable to your shell settings (e.g. .bashrc) instead of the above method
export PIP_INDEX_URL=http://localhost:3141/root/pypi/+simple/pip install
Now you have seen how to setup a local PyPI mirror server using DevPi.
Thank You For Your Attention
Jesus Saves
By Jesse E.Agbe(JCharis)