Python 3

Contents view

Link to setup python environment !

Launching python 3

For using Python, typ the command “python3”

$ python3
Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.

You can use a more recent version of Python with the loading of a Python module. For example, 

$ module load programming/python3/3.10.4
$ python3
Python 3.10.4 (main, Sep 13 2022, 15:31:07) [GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

Loading a library.

$ python3
Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas

A library can be installed in a Python version but not in another.

$ module load programming/python3/3.10.4
$ python3
Python 3.10.4 (main, Sep 13 2022, 15:31:07) [GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'

The library called pandas is well installed in the version 3.6.8 but not into the version 3.10.4.

Installation of a library

As a user, possibility to install a Python library.

Example with pandas

$ ml compiler/python/3.11.0 
$ pip3.11 install --no-cache-dir --upgrade --user pandas
Collecting pandas
  Downloading pandas-1.5.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.0/12.0 MB 122.9 MB/s eta 0:00:00
Collecting python-dateutil>=2.8.1
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 281.3 MB/s eta 0:00:00
Collecting pytz>=2020.1
  Downloading pytz-2022.7.1-py2.py3-none-any.whl (499 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 499.4/499.4 kB 300.8 MB/s eta 0:00:00
Collecting numpy>=1.21.0
  Downloading numpy-1.24.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 150.2 MB/s eta 0:00:00
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: pytz, six, numpy, python-dateutil, pandas
Successfully installed numpy-1.24.2 pandas-1.5.3 python-dateutil-2.8.2 pytz-2022.7.1 six-1.16.0
$ python3
Python 3.11.0 (main, Dec  1 2022, 09:03:09) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas

In some cases, installation may fail. Check that the library is compatible with your version of Python.

Example with ktrain

For example, at the time of writing, the ktrain library fails with Python 3.11.2.

$ ml compiler/python/3.11.2 
$ python3 --version
Python 3.11.2
$ pip3 install --user ktrain
Collecting ktrain
  Using cached ktrain-0.37.2.tar.gz (25.3 MB)
...
src/cchardet/_cchardet.cpp:196:12: fatal error: longintrepr.h: No such file or directory

To install it, you need an older version of Python. For example, Python 3.9.16.

$ ml load compiler/python/3.9.16
$ python3 --version
Python 3.9.16
$ pip install --upgrade pip
$ pip3 install --upgrade --user ktrain

However, you also need to make sure that the import works.

$ python3
Python 3.9.16 (main, Mar  9 2023, 23:53:30) 
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ktrain
/.../.local/lib/python3.9/site-packages/ktrain/imports.py:59: UserWarning: TensorFlow is not installed and will be needed if training neural networks, but non-TensorFlow features in ktrain can still be used. See https://github.com/amaiya/ktrain/blob/master/README.md
  warnings.warn(TF_WARNING)
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../.local/lib/python3.9/site-packages/ktrain/__init__.py", line 2, in <module>
    from . import utils as U
  File "/.../.local/lib/python3.9/site-packages/ktrain/utils.py", line 30, in <module>
    if version.parse(tf.__version__) < version.parse("2.11")
AttributeError: 'NoneType' object has no attribute '__version__'

In our example, the error is corrected by installing at least the tensorflow package.

$ pip install --upgrade torch
$ pip install --upgrade tensorflow

And now

$ python3
Python 3.9.16 (main, Mar  9 2023, 23:53:30) 
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ktrain
>>> quit()

Example with pytorch

The following script installs locally pytorch (Be careful! The script uninstalls it once the task is done, too).
The script of test my_torch.py can be found at : https://pytorch.org/get-started/locally/

Script of test

from __future__ import print_function
import torch
x = torch.rand(5, 3)
print(x)

Script for launching the job pytorch

Installation pytorch, running the program then uninstallation of pytorch

#!/bin/sh
#SBATCH --job-name=pytorch
#SBATCH --partition=normal
#SBATCH --output=job-%j.out
#SBATCH --nodes=1
ml compiler/python/3.11.2
mkdir -p ~/tmp
export TMPDIR=~/tmp
echo "--- INSTALL PYTHON LIBRARIES ---"
pip3 install --no-cache-dir --upgrade --user torch torchvision
echo "--- LAUNCH PROGRAM ---"
date
time python3 my_torch.py
date
echo "--- DESINSTALLATION ---" /usr/bin/yes | pip3 uninstall torchvision
/usr/bin/yes | pip3 uninstall torch

Notes :

  1. For installing locally a python library on his/her account as a simple user, we add the option –user in the command pip3.
  2. The libraries are put into your hidden folder .local.