Setting up python virtual environment

 

Warning! Python 2 has reached the end of life on January 2020, after many years of prior notifications. It is not supported in any way. Do not use it and do not ask about it.

 

It is not possible to provide a single, system-wide python installation because of various versions of packages, which can be conflicting. Thus the "system python" is installed for system tools rather then for users - if it fits your needs use it, but do not ask to install additional packages. The proper way to use python are user's virtual environments. They enable multiple, independent python installations, each with own set of packages.

There are two major methods to create the python virtual environment:

  • python built-in virtual environment with pip package manager - simple to setup and use, limited to python packages, dependencies are sometimes problematic, packages are compiled locally so quite often additional packages and specific compilers are needed;
  • conda package manager - more advanced and not limited to python packages only, for example when installing tensorflow-gpu it will install cuda toolkit and cuDNN library too. Precompiled binaries are distributed and the dependency handling is much better. Conda is part of anaconda (large distribution of python packages - not recommended since probably you won't use more than 1% of it) and miniconda (minimal python distribution - recommended) distributions. In general conda is recommended to use on the chuck cluster.

 

Below we present both methods tested on the chuck cluster. 

conda (using miniconda3)

 

  1. Install miniconda3 according to : https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html  e.g.:
      
    cd /work/chuck/$USER
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
    It comes with the most current version of python. For older versions check https://docs.conda.io/en/latest/miniconda.html
    Tip 1. Do not install in /home , change the installation location to /work/chuck/... e.g. /work/chuck/pci/miniconda3
    Tip 2. Choose to initialize Miniconda3
    Tip 3. Initilization modifies your ~/.bashrc - it will work on desktops but not on chuck due to differences between Ubuntu and Redhat based systems. To make it working on chuck add this line at the end of your ~/.profile file:

    [[ -r ~/.bashrc ]] && source ~/.bashrc

     Tip 4. If you do not want to automatically activate the default environment on each shell startup (which results in a small delay) execute this (recommended):

    conda config --set auto_activate_base false

     

  2. Test the installation. The changes made above will work in a new shells - open a new terminal and check installed packages:

    conda list
     

  3. Add conda-forge repository (it provides more and more recent versions of many packages)

    conda config --add channels conda-forge
     

  4. Create a new virtual environment

    conda create -n myenv python=3 pandas matplotlib jupyter nb_conda
     

  5. Activate the environment

    conda activate myenv
     

  6. To install additional packages, e.g. numpy  use

    conda install numpy

 

 For details see the documentation: https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#

 

Virtual environment with Python 3.4.5

 

Python 3 comes with virtual environment tool built-in (pyvenv).

Optional! If you need to compile specific python 3 version.

cd /work/chuck/$USER
mkdir -p .pythons
cd .pythons
wget https://www.python.org/ftp/python/3.4.5/Python-3.4.5.tgz
tar zxvf Python-3.4.5.tgz
cd Python-3.4.5
make clean
mkdir /work/chuck/$USER/.pythons/3.4
./configure --prefix=/work/chuck/$USER/.pythons/3.4
make -j4
make install
cd ..
rm -rf Python-3.4.5*


Now let's create a virtual environment.

cd /work/chuck/$USER
mkdir python-virtualenvs
cd python-virtualenvs
/work/chuck/$USER/.pythons/3.4/bin/pyvenv myenv3.4
# since python 3.5 the last line is changed to:
/work/chuck/$USER/.pythons/3.4/bin/python3 -m venv  myenv_name


To work in the specific virtual environment issue command
     source /work/chuck/$USER/python-virtualenvs/myenv3.4/bin/activate
(to leave type deactivate )

To install specific packages use pip in active virtual environment e.g.:

pip install numpy
pip install scipy
pip install matplotlib
pip install pandas


To list installed packages type:

pip freeze