Setting up python virtual environment

 

Python installed in the system is actually for system tools. If you need additional packages and/or specific versions you have to use own python virtual environment. It enables you to have multiple, independent python installations, each with own set of packages. It is the recommended way of using python at CAMK. Examples below show how to prepare the python virtual environment with python 2.x and python 3.x and install some packages. They were tested on the chuck cluster.


 

Python 2.7 example

 

Login to chuck.

 

Optional! If you will always use devtoolset-7 environment you can use it also to compile python. Enable it with

scl enable devtoolset-7 bash

Remember to do it every time before python is used - you can put it in the shell init script.

 

Optional! Now we compile own python. If you are OK with the version installed in the system (2.7.5) skip this step.

cd
mkdir -p .pythons/2.7-devtoolset7
cd .pythons
wget https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tgz
tar zxvf Python-2.7.12.tgz
cd Python-2.7.12
./configure --prefix=$HOME/.pythons/2.7-devtoolset7
make -j10
make install
cd ..
rm -rf Python-2.7.12*


At this point your own python 2.7 installation is in $HOME/.pythons/2.7-devtoolset7.

 

Now we will create a virtual environment.

cd $HOME
mkdir python-virtualenvs
cd python-virtualenvs

If you compiled own python type:

virtualenv myenv2.7-devt7 --python=$HOME/.pythons/2.7-devtoolset7/bin/python

If you want to use the system installed python type:

virtualenv myenv2.7 --python=python2.7

(you can have both versions, just use different names).


To work in a specific virtual environment issue command
     source ~/python-virtualenvs/myenv2.7/bin/activate
and to leave it type deactivate .

To install specific packages in the active virtual environment use pip e.g.:

pip install setuptools --upgrade    #do this to avoid problems with installation of certain packages like matplotlib
pip install numpy

pip install scipy
pip install matplotlib
pip install pandas


To list installed packages type:

pip freeze


Python 3.4.5 example

 

The procedure for python 3 is almost the same. The only difference is that python 3 comes with virtual environment tool built-in (pyvenv).

Optional!

cd
mkdir -p .pythons/3.4
cd .pythons
wget https://www.python.org/ftp/python/3.4.5/Python-3.4.5.tgz
tar zxvf Python-3.4.5.tgz
cd Python-3.4.5
make clean
./configure --prefix=$HOME/.pythons/3.4
make -j4
make install
cd ..
rm -rf Python-3.4.5*


Now let's create virtual environment.

cd $HOME
mkdir python-virtualenvs
cd python-virtualenvs
~/.pythons/3.4/bin/pyvenv myenv3.4


To work in specific virtual environment issue command
     source ~/python-virtualenvs/myenv3.4/bin/activate
(to leave type deactivate )

To install specific packages use pip in active virtual environment e.g.:

pip install numpy
pip install scipy
pip install matplotlib
pip install pandas


To list installed packages type:

pip freeze