Setting up python virtual environment

 

Warning! Python 2 has reached the end of life on January 2020, after many years of prior notifications. It is not supported in any way. Do not use it.

 

It is not possible to have a single, system-wide python installation due to many conflicting versions of packages. System python is to be used by the system tools - if it fits your needs use it, but do not ask to install additional packages. The correct way to use python are virtual environments. They enable you to have multiple, independent python installations, each with own set of packages.

There are two major methods to create python virtual environment:

  • python built-in virtual environment with pip package manager - simple to setup and use, limited to python packages, dependencies are sometimes problematic,
  • conda package manager - much more advanced, is not limited to python packages only, for example when installing tensorflow-gpu it will install also cuda toolkit and cuDNN library. It's part of anaconda (large distribution of python packages - not recommended since probably you won't use more than 1% of it) and miniconda (minimal python distribution - recommended) distributions.

 

Below we present both methods tested on the chuck cluster.


 

Virtual environment with Python 3.4.5

 

Python 3 comes with virtual environment tool built-in (pyvenv).

Optional! If you need to compile specific python 3 version.

cd
mkdir -p .pythons/3.4
cd .pythons
wget https://www.python.org/ftp/python/3.4.5/Python-3.4.5.tgz
tar zxvf Python-3.4.5.tgz
cd Python-3.4.5
make clean
./configure --prefix=$HOME/.pythons/3.4
make -j4
make install
cd ..
rm -rf Python-3.4.5*


Now let's create virtual environment.

cd $HOME
mkdir python-virtualenvs
cd python-virtualenvs
~/.pythons/3.4/bin/pyvenv myenv3.4
# since python 3.5 the last line is changed to:
python3 -m venv  myenv_name


To work in the specific virtual environment issue command
     source ~/python-virtualenvs/myenv3.4/bin/activate
(to leave type deactivate )

To install specific packages use pip in active virtual environment e.g.:

pip install numpy
pip install scipy
pip install matplotlib
pip install pandas


To list installed packages type:

pip freeze

 

conda (using miniconda3)

 

  1. Install miniconda3 according to : https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html  e.g.:

    cd /work/chuck/pci
    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
    bash Miniconda3-latest-Linux-x86_64.sh

    Tip 1. Do not install in /home , change  installation location to /work e.g. /work/chuck/pci/miniconda3
    Tip 2. Choose to initialize Miniconda3
    Tip 3. Initilization modifies your ~/.bashrc - it will work on desktops but not on chuck due to differences between Ubuntu and Redhat based systems. To make it working on chuck add this line at the end of your ~/.profile file:

    [[ -r ~/.bashrc ]] && source ~/.bashrc

     Tip 4. We recommend to activate conda environment manually - execute this to disable automatic loading on shell startup:

    conda config --set auto_activate_base false


  2. Test the installation. The changes made above will work in a new shells - open a new terminal and check installed packages:

    conda list

  3. Add conda-forge repository

    conda config --add channels conda-forge

  4. Create a new virtual environment

    conda create -n myenv python=3 pandas matplotlib jupyter nb_conda

  5. Activate the environment

    conda activate myenv

 

 For details see the documentation: https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#