Setting up python virtual environment

 

Warning! Python 2 has reached the end of life on January 2020, after many years of prior notifications. It is not supported in any way. Do not use it.

 

It is not possible to have a single, system-wide python installation due to many conflicting versions of packages. System python is to be used by the system tools - if it fits your needs use it, but do not ask to install additional packages. The proper way to use python are virtual environments. They allow you to have multiple, independent python installations, each with own set of packages.

There are two major methods to create python virtual environment:

• python built-in virtual environment with pip package manager - simple to setup and use, limited to python packages, dependencies are sometimes problematic,
• conda package manager - more advanced and not limited to python packages only, for example when installing tensorflow-gpu it will install cuda toolkit and cuDNN library too. Conda is part of anaconda (large distribution of python packages - not recommended since probably you won't use more than 1% of it) and miniconda (minimal python distribution - recommended) distributions. In general conda is recommended to use on chuck.

 

Below we present both methods tested on the chuck cluster. 

---

 

Virtual environment with Python 3.4.5

 

Python 3 comes with virtual environment tool built-in (pyvenv).

Optional! If you need to compile specific python 3 version.

cd
mkdir -p .pythons/3.4
cd .pythons
wget https://www.python.org/ftp/python/3.4.5/Python-3.4.5.tgz
tar zxvf Python-3.4.5.tgz
cd Python-3.4.5
make clean
./configure --prefix=$HOME/.pythons/3.4
make -j4
make install
cd ..
rm -rf Python-3.4.5*

Now let's create virtual environment.

cd $HOME
mkdir python-virtualenvs
cd python-virtualenvs
~/.pythons/3.4/bin/pyvenv myenv3.4
# since python 3.5 the last line is changed to:
python3 -m venv  myenv_name

To work in the specific virtual environment issue command
     source ~/python-virtualenvs/myenv3.4/bin/activate
(to leave type deactivate )

To install specific packages use pip in active virtual environment e.g.:

pip install numpy
pip install scipy
pip install matplotlib
pip install pandas

To list installed packages type:

pip freeze

conda (using miniconda3)

 

1. Install miniconda3 according to : https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html  e.g.:
   
   

   cd /work/chuck/pci
   wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
   bashMiniconda3-latest-Linux-x86_64.sh
   
   

   Tip 1. Do not install in /home , change installation location to /work/chuck/... e.g. /work/chuck/pci/miniconda3
   Tip 2. Choose to initialize Miniconda3
   Tip 3. Initilization modifies your ~/.bashrc - it will work on desktops but not on chuck due to differences between Ubuntu and Redhat based systems. To make it working on chuck add this line at the end of your ~/.profile file:

   [[ -r ~/.bashrc ]] && source ~/.bashrc

    Tip 4. We recommend to activate conda environment manually - execute this to disable automatic loading on shell startup:

   conda config --set auto_activate_base false
   
   
   

2. Test the installation. The changes made above will work in a new shells - open a new terminal and check installed packages:
   
   conda list
   
   

3. Add conda-forge repository
   
   conda config --add channels conda-forge
   
   

4. Create a new virtual environment
   
   conda create -n myenv python=3 pandas matplotlib jupyter nb_conda
   
   

5. Activate the environment
   
   conda activate myenv
   
   

6. To install additional packages, e.g. numpy  use
   
   conda install numpy

 For details see the documentation: https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#