Setting up python virtual environment

Warning! Python 2 has reached the end of life on January 2020, after many years of prior notifications. It is not supported in any way. Do not use it and do not ask about it.

It is not possible to provide a single, system-wide python installation because of various versions of packages, which can be conflicting. Thus the "system python" is installed for system tools rather then for users - if it fits your needs use it, but do not ask to install additional packages. The proper way to use python are user's virtual environments. They enable multiple, independent python installations, each with own set of packages.

There are two major methods to create the python virtual environment:

python built-in virtual environment with pip package manager - simple to setup and use, limited to python packages, dependencies are sometimes problematic, packages are compiled locally so quite often additional packages and specific compilers are needed;
conda package manager - more advanced and not limited to python packages only, for example when installing tensorflow-gpu it will install cuda toolkit and cuDNN library too. Precompiled binaries are distributed and the dependency handling is much better. Conda is part of anaconda (large distribution of python packages - not recommended since probably you won't use more than 1% of it) and miniconda (minimal python distribution - recommended) distributions. In general conda is recommended to use on the chuck cluster.

Below we present both methods tested on the chuck cluster.

conda/mamba from miniforge3 package

Mamba is a faster version of conda (written in C++), usually you can just substitute mamba whenever you would use conda.
Miniforge3 package comes with conda-forge preconfigured as the default channel.

Install miniforge3 according to https://github.com/conda-forge/miniforge e.g.:
```
cd /work/$USER
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh
```
It comes with the most current version of python.
Tip 1. Do not install in /home (too big) or /work/chuck (too many small files), change the installation location to /work/$USER/miniforge3
Tip 2. Choose to update your shell to initialize conda
Tip 3. Initilization modifies your ~/.bashrc - it will work on desktops but not on chuck due to differences between Ubuntu and Redhat based systems. To make it working on chuck add this line at the end of your ~/.profile file:

[[ -r ~/.bashrc ]] && source ~/.bashrc

Tip 4. If you do not want to automatically activate the default environment everytime you start a new shell (which incures some delay) execute this (recommended):

conda config --set auto_activate_base false
The changes made above will work in a new shells - open a new terminal.
Optionally update base environment:

mamba update --all
Test the installation:

mamba list
Create a new virtual environment. It is not recommended to install anything else into base environment!

mamba create -n myenv pandas matplotlib jupyter
Activate the environment

mamba activate myenv
To install additional packages, e.g. numpy use

mamba install numpy

For details see the documentation: https://conda-forge.org/docs/

conda (using miniconda3)

Install miniconda3 according to : https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html e.g.:
```
cd /work/chuck/$USER
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
```
It comes with the most current version of python. For older versions check https://docs.conda.io/en/latest/miniconda.html
Tip 1. Do not install in /home , change the installation location to /work/chuck/... e.g. /work/chuck/pci/miniconda3
Tip 2. Choose to initialize Miniconda3
Tip 3. Initilization modifies your ~/.bashrc - it will work on desktops but not on chuck due to differences between Ubuntu and Redhat based systems. To make it working on chuck add this line at the end of your ~/.profile file:

[[ -r ~/.bashrc ]] && source ~/.bashrc

Tip 4. If you do not want to automatically activate the default environment on each shell startup (which results in a small delay) execute this (recommended):

conda config --set auto_activate_base false
Test the installation. The changes made above will work in a new shells - open a new terminal and check installed packages:

conda list
Add conda-forge repository (it provides more and more recent versions of many packages)

conda config --add channels conda-forge
Create a new virtual environment

conda create -n myenv python=3 pandas matplotlib jupyter nb_conda
Activate the environment

conda activate myenv
To install additional packages, e.g. numpy use

conda install numpy

For details see the documentation: https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#

Virtual environment with Python 3.4.5

Python 3 comes with virtual environment tool built-in (pyvenv).

Optional! If you need to compile specific python 3 version.

cd /work/chuck/$USER
mkdir -p .pythons
cd .pythons
wget https://www.python.org/ftp/python/3.4.5/Python-3.4.5.tgz
tar zxvf Python-3.4.5.tgz
cd Python-3.4.5
make clean mkdir /work/chuck/$USER/.pythons/3.4
./configure --prefix=/work/chuck/$USER/.pythons/3.4
make -j4
make install
cd ..
rm -rf Python-3.4.5*

Now let's create a virtual environment.

cd /work/chuck/$USER
mkdir python-virtualenvs
cd python-virtualenvs
/work/chuck/$USER/.pythons/3.4/bin/pyvenv myenv3.4 # since python 3.5 the last line is changed to: /work/chuck/$USER/.pythons/3.4/bin/python3 -m venv myenv_name

To work in the specific virtual environment issue command
source /work/chuck/$USER/python-virtualenvs/myenv3.4/bin/activate
(to leave type deactivate )

To install specific packages use pip in active virtual environment e.g.:

pip install numpy
pip install scipy
pip install matplotlib
pip install pandas

To list installed packages type:

pip freeze