NewETAPE projectEmuframe Tape Server (ETAPE) project
Python
This page was auto-translated from Spanish

Python on z/OS: virtual environment (venv and pyproject.toml)

This article continues the Python on z/OS series: Update Python on z/OS, pySDSF on z/OS 3.2: watch the spool and alert on Microsoft Teams and Create a user for Python batch scripts (RACF).

Until now, the SPOOLCHK job runs python3, the one it finds in the PATH through the /u/python/current link. That link is handy for switching versions, but it has a problem: pySDSF is installed in 3.13 (its wheel is cp313), and if tomorrow I point current to 3.14, the job stops working without anyone touching it.

The solution is a virtual environment: a folder that uses a specific Python version and has its own packages. And with a pyproject.toml, the script is installed as a command, spool-alert, which is what the job runs.

What a virtual environment and pyproject.toml are

  • A virtual environment (venv) is a directory with a lightweight Python that uses the interpreter it was created with and has its own package folder. What I install in it does not touch the system Python or other users' Python. Its pyvenv.cfg file records the path of the Python that created it.
  • pyproject.toml is the standard file that describes a Python project: its name, its version, the Python version it needs, its dependencies and the commands it creates when installed.

The Python versions I have

On my system these live side by side:

  • The 3.14 of the stock image, in /usr/lpp/IBM/cyp/v3r14/pyz, which I do not touch.
  • The 3.13.15.1 and the 3.14.7.1 that I installed, each in its own zFS under /u/python.
  • The /u/python/current link, which points to the one I use day to day.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 1

I create a fixed link for 3.13, /u/python/py313, to have a short path to this version that does not change when I point current to another one. I use it to create the environment, in the LIBPATH of the jobs and to find the setuptools wheel. And if I install a new 3.13 release, I only need to point py313 to it and create the environment again, without touching the jobs. I create it with a job submitted by IBMUSER:

//PYLINK   JOB (ACCT),'PYTHON LINK',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID
//* Create a fixed link to Python 3.13, apart from the current link
//RUN      EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH cd /u/python &&
rm -f py313 &&
ln -s python-v3.13.15.1/usr/lpp/IBM/cyp/v3r13/pyz py313 &&
ls -l /u/python
/*

The link is relative, like current: it points to the 3.13 folder inside /u/python. The rm -f removes the link if it already existed, so the job can be run again.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 2

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 3

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 4

Create the environment as PYBATCH

The environment belongs to PYBATCH, so a job with USER=PYBATCH creates it, and it lives in its HOME, in /u/pybatch/venv. In the same job I install pySDSF in the environment from its wheel, which comes with SDSF in /usr/lpp/sdsf/python/dist:

//PYVENV   JOB (ACCT),'PYBATCH VENV',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID,
//             USER=PYBATCH,REGION=256M
//* Create a Python 3.13 virtual environment for PYBATCH and
//* install pySDSF in it
//RUN      EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH PY=/u/python/py313;
export LIBPATH=$PY/lib:$LIBPATH;
V=/u/pybatch/venv;
W=/usr/lpp/sdsf/python/dist;
$PY/bin/python3 -m venv $V &&
$V/bin/pip install --no-index $W/pysdsf-3.2-cp313-none-any.whl &&
$V/bin/pip list &&
cat $V/pyvenv.cfg
/*

Step by step:

  • PY is 3.13, through its fixed link. IBM asks for the Python lib to be in the LIBPATH, and the PYBATCH .profile sets the one of current; that is why I put the 3.13 one first, so it is found first even if current points to another version.
  • python3 -m venv creates the environment with 3.13.
  • pip install --no-index installs the pySDSF wheel without looking for anything on the internet.
  • pip list and pyvenv.cfg show what is in the environment and which Python created it.

I write each complete command on its own STDPARM line, ending with ; or &&, so that none is split across two lines.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 5

We can see the processes that run to create the environment.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 6

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 7

We check that it ends fine.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 8

We see that it installed the pySDSF 3.2 package.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 9

In pyvenv.cfg, command shows that the environment was created with /u/python/py313, but home and executable point to the real path of 3.13.15.1: venv resolves the link when it creates the environment, and the environment is tied to that release. If one day I install another 3.13 release and point py313 to it, the environment would keep using the old one, so it has to be created again: delete /u/pybatch/venv and run PYVENV and PYINST.

Test the environment with who()

I run who.py with the Python of the environment:

//PYWHO    JOB (ACCT),'PYBATCH WHO',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID,
//             USER=PYBATCH,REGION=256M
//* Run who.py with the Python of the PYBATCH virtual environment
//RUN      EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH export LIBPATH=/u/python/py313/lib:$LIBPATH;
/u/pybatch/venv/bin/python /u/pybatch/scripts/who.py
/*

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 10

We confirm that it runs correctly. Also, we are using the PYBATCH user, which we created in Create a user for Python batch scripts (RACF), so we should not run into problems.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 11

The project with pyproject.toml

I create the project folder, /u/pybatch/spoolalert, with the script and its pyproject.toml:

[build-system]
requires = ["setuptools>=70"]
build-backend = "setuptools.build_meta"

[project]
name = "spool-alert"
version = "1.0.0"
requires-python = ">=3.13,<3.14"
dependencies = ["pysdsf"]

[project.scripts]
spool-alert = "spool_alert:main"

[tool.setuptools]
py-modules = ["spool_alert"]
  • requires-python records that the project is for 3.13: if someone installs it with another version, pip rejects it.
  • dependencies says that it needs pySDSF, which is already in the environment.
  • [project.scripts] creates the spool-alert command, which calls the main() function of spool_alert.py.

To install the project, pip needs setuptools, and a new environment does not include it. There is no need to download it: IBM's Python SDK ships its wheel among the Python test data, in lib/python3.13/test/wheeldata (setuptools-80.8.0-py3-none-any.whl). It is pure Python (py3-none-any), so it works on z/OS. If it were not there, it can be downloaded from PyPI and uploaded in binary.

I write the pyproject.toml with a job: IEBGENER copies the lines in the JCL to a z/OS UNIX file. (There are more ways to create the file; this is simply a different way of doing it, to have more resources we can use in different situations.)

//PYTOML   JOB (ACCT),'PYBATCH TOML',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID
//* Write pyproject.toml from the in-stream lines (EBCDIC)
//TOML     EXEC PGM=IEBGENER
//SYSPRINT DD SYSOUT=*
//SYSIN    DD DUMMY
//SYSUT2   DD PATH='/tmp/pyproject.e',
//            PATHOPTS=(OWRONLY,OCREAT,OTRUNC),
//            PATHMODE=(SIRUSR,SIWUSR,SIRGRP,SIROTH),
//            FILEDATA=TEXT,RECFM=FB,LRECL=80,BLKSIZE=3200
//SYSUT1   DD *
[build-system]
requires = ["setuptools>=70"]
build-backend = "setuptools.build_meta"

[project]
name = "spool-alert"
version = "1.0.0"
requires-python = ">=3.13,<3.14"
dependencies = ["pysdsf"]

[project.scripts]
spool-alert = "spool_alert:main"

[tool.setuptools]
py-modules = ["spool_alert"]
/*
  • SYSUT1 holds the content of the pyproject.toml as is, between DD * and /*.
  • SYSUT2, with PATH, writes to a z/OS UNIX file instead of a data set. PATHOPTS creates it, or empties it if it already exists, and PATHMODE gives it rw-r--r-- permissions.
  • FILEDATA=TEXT makes each line end with a newline. The lines stay in EBCDIC, as they are in the JCL, in a temporary file, /tmp/pyproject.e.

The emulator code page. The JCL lines get the code page of the 3270 emulator I use to type or paste them in the editor, and the next job converts them from IBM-1047. With another code page, for example the Spanish one (IBM-284), the brackets of the pyproject.toml get other codes, iconv converts them as if they were 1047 and pip fails to read the file with a UnicodeDecodeError. That is why I set the emulator session to 1047 before typing or pasting the job. And with NUM OFF in the editor, so there are no sequence numbers in columns 73 to 80, which would end up in the file.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 12

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 13

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 14

Python expects the file in ASCII. Another job, also submitted by IBMUSER:

  • Creates the project folder and the wheels folder.
  • Converts the file to ASCII and deletes the temporary one.
  • Copies the script and the setuptools wheel.
  • Sets the text tag on the project files, like on the scripts (the wheel is binary and does not get it).
  • Sets the owner to PYBATCH.
//PYPROJ   JOB (ACCT),'PYBATCH PROJ',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID
//* Create the spool-alert project and the wheels folder for PYBATCH
//RUN      EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH P=/u/pybatch/spoolalert;
WH=/u/pybatch/wheels;
T=/u/python/py313/lib/python3.13/test/wheeldata;
mkdir -p $P $WH &&
iconv -f IBM-1047 -t ISO8859-1 /tmp/pyproject.e > $P/pyproject.toml &&
rm /tmp/pyproject.e &&
cp /u/pybatch/scripts/spool_alert.py $P/ &&
cp $T/setuptools-*.whl $WH/ &&
chtag -tc ISO8859-1 $P/* &&
chown -R PYBATCH:PYGRP $P $WH &&
ls -lT $P $WH
/*

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 15

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 16

And I install it in the environment with another job as PYBATCH:

//PYINST   JOB (ACCT),'PYBATCH INSTALL',CLASS=A,MSGCLASS=X,
//             NOTIFY=&SYSUID,USER=PYBATCH,REGION=256M
//* Install the spool-alert project in the PYBATCH virtual environment
//RUN      EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH export LIBPATH=/u/python/py313/lib:$LIBPATH;
V=/u/pybatch/venv;
WH=/u/pybatch/wheels;
P=/u/pybatch/spoolalert;
$V/bin/pip install --no-index --find-links $WH setuptools &&
$V/bin/pip install --no-index --no-build-isolation $P &&
ls -l $V/bin
/*
  • The first pip install installs setuptools from the wheels folder.
  • The second one installs the project. With --no-build-isolation, pip uses the setuptools of the environment instead of trying to download it.
  • ls -l shows the spool-alert command in bin.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 17

We wait for it to end.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 18

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 19

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 20

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 21

The watch job with the command

The SPOOLCHK job now looks like this:

//SPOOLCHK JOB (ACCT),'SPOOL CHECK',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID,
//             USER=PYBATCH,REGION=256M
//* Check JES2 spool usage with pySDSF and alert Microsoft Teams
//RUN      EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH export LIBPATH=/u/python/py313/lib:$LIBPATH;
/u/pybatch/venv/bin/spool-alert
/*

The PYBATCH .profile still loads the webhook URL, so nothing else changes in the job.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 22

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 23

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 24

The spool is below 70 %, so there is no alert. To test the card, I lower the threshold with the SPOOL_THRESHOLD variable:

//SPOOLCHK JOB (ACCT),'SPOOL CHECK',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID,
//             USER=PYBATCH,REGION=256M
//* Check JES2 spool usage with a 20% threshold to test the alert
//RUN      EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH export LIBPATH=/u/python/py313/lib:$LIBPATH;
SPOOL_THRESHOLD=20 /u/pybatch/venv/bin/spool-alert
/*

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 25

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 26

We check that the alert arrived in Microsoft Teams.

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 27

More than one command in the same project

A project can create several commands. Each line of [project.scripts] is a command, in the form name = "module:function", and each new module is added to py-modules. For example, to also have who.py as a command, I first rewrite it with a main() function:

"""Show the SDSF session of the user (who) with pySDSF."""
import sys

from pysdsf import Connection
from pysdsf.isfexception import ISFException


def main():
    connection = Connection()
    try:
        for row in connection.who():
            print(row)
    except ISFException as e:
        print(e, file=sys.stderr)
        return 8
    finally:
        connection.close()
    return 0


if __name__ == "__main__":
    sys.exit(main())
  • It does the same as before, but if SDSF returns an error it writes it to STDERR and returns 8, so the job step no longer ends with RC 0.
  • With the if __name__ at the end it can still be run as a script, as in the PYWHO job.

I upload it in binary to /u/pybatch/scripts, replacing the old one, and, as IBMUSER from OMVS, I set its tag and give it to PYBATCH:

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 28

chtag -tc ISO8859-1 /u/pybatch/scripts/who.py
chown PYBATCH:PYGRP /u/pybatch/scripts/who.py
ls -lT /u/pybatch/scripts/who.py

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 29

In the pyproject.toml I raise the version, add the command and add the module, and I write it again with PYTOML1, with the emulator session in 1047:

//PYTOML1  JOB (ACCT),'PYBATCH TOML',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID
//* Write pyproject.toml from the in-stream lines (EBCDIC)
//TOML     EXEC PGM=IEBGENER
//SYSPRINT DD SYSOUT=*
//SYSIN    DD DUMMY
//SYSUT2   DD PATH='/tmp/pyproject.e',
//            PATHOPTS=(OWRONLY,OCREAT,OTRUNC),
//            PATHMODE=(SIRUSR,SIWUSR,SIRGRP,SIROTH),
//            FILEDATA=TEXT,RECFM=FB,LRECL=80,BLKSIZE=3200
//SYSUT1   DD *
[build-system]
requires = ["setuptools>=70"]
build-backend = "setuptools.build_meta"

[project]
name = "spool-alert"
version = "1.1.0"
requires-python = ">=3.13,<3.14"
dependencies = ["pysdsf"]

[project.scripts]
spool-alert = "spool_alert:main"
spool-who = "who:main"

[tool.setuptools]
py-modules = ["spool_alert", "who"]
/*

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 30

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 31

And PYPROJ1, which now also copies who.py to the project:

//PYPROJ1  JOB (ACCT),'PYBATCH PROJ',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID
//* Copy who.py and the new pyproject.toml to the spool-alert project
//RUN      EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH P=/u/pybatch/spoolalert;
WH=/u/pybatch/wheels;
T=/u/python/py313/lib/python3.13/test/wheeldata;
mkdir -p $P $WH &&
iconv -f IBM-1047 -t ISO8859-1 /tmp/pyproject.e > $P/pyproject.toml &&
rm /tmp/pyproject.e &&
cp /u/pybatch/scripts/spool_alert.py /u/pybatch/scripts/who.py $P/ &&
cp $T/setuptools-*.whl $WH/ &&
chtag -tc ISO8859-1 $P/* &&
chown -R PYBATCH:PYGRP $P $WH &&
ls -lT $P $WH
/*

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 32

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 33

Then I run PYINST again, unchanged, as PYINST1: it installs version 1.1.0 and spool-who appears in bin next to spool-alert.

//PYINST1  JOB (ACCT),'PYBATCH INSTALL',CLASS=A,MSGCLASS=X,
//             NOTIFY=&SYSUID,USER=PYBATCH,REGION=256M
//* Install the spool-alert project in the PYBATCH virtual environment
//RUN      EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH export LIBPATH=/u/python/py313/lib:$LIBPATH;
V=/u/pybatch/venv;
WH=/u/pybatch/wheels;
P=/u/pybatch/spoolalert;
$V/bin/pip install --no-index --find-links $WH setuptools &&
$V/bin/pip install --no-index --no-build-isolation $P &&
ls -l $V/bin
/*

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 34

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 35

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 36

And I run it with a job as PYBATCH:

//PYWHO1   JOB (ACCT),'PYBATCH WHO',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID,
//             USER=PYBATCH,REGION=256M
//* Run the spool-who command of the spool-alert project
//RUN      EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH export LIBPATH=/u/python/py313/lib:$LIBPATH;
/u/pybatch/venv/bin/spool-who
/*

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 37

Python on z/OS: virtual environment (venv and pyproject.toml) – screenshot 38

A few more things about the commands:

  • The command calls the function without arguments. If it needs parameters, it reads them with sys.argv or argparse.
  • What the function returns is the exit code of the command: if it is not 0, the BPXBATCH step does not end with RC 0.
  • One file can provide several commands, each with a different function: for example, spool-alert = "spool_alert:main" and spool-status = "spool_alert:status", if spool_alert.py had a status() function.
  • pip copies the modules into the environment, in its site-packages. If I change a script, I have to run PYTOML1, PYPROJ1 and PYINST1 again, in that order, for the command to use the new version: PYPROJ1 needs the file that PYTOML1 leaves in /tmp.
  • Raising the project version in the pyproject.toml (the version line, from 1.0.0 to 1.1.0) is not required: pip reinstalls the project from its folder even if it has the same version. I raise it to keep a record of the change.

Loose modules or a package

In Python, each .py file is a module, and its name is the file name without the .py: who.py is the who module. When a program runs import who, Python looks for a module with that name in the folders it knows. In a virtual environment, the main one is site-packages, which is where pip puts everything it installs.

What we have done in this project are loose modules: spool_alert.py and who.py sit directly in the project folder, and when it is installed pip copies them as they are to site-packages, side by side and next to what other packages install, such as pySDSF.

A package is a folder that groups several modules under one name. For Python to treat it as a package, it contains an __init__.py file, which can be empty. With a package, the project would look like this:

/u/pybatch/spoolalert/
  pyproject.toml
  spoolalert/
    __init__.py
    alert.py
    who.py

Now the module is no longer just who, but spoolalert.who: the package name, a dot and the module name. In the pyproject.toml, py-modules is replaced by packages = ["spoolalert"], and the commands point to "spoolalert.alert:main" and "spoolalert.who:main".

What changes:

  • Names do not clash. who is a very common name: if another package installed in the environment also brought a who.py, one would overwrite the other in site-packages. Inside the package it is spoolalert.who, and there is no confusion.
  • Sharing code is easy. If both commands need the same thing, for example opening the SDSF connection or sending the alert to Teams, it goes in a module of the package, such as spoolalert/common.py, and both use it, instead of repeating it in each script.
  • Everything stays together. With many scripts, site-packages has a single folder for the project instead of many loose files.

With two scripts, as now, loose modules are enough. If the project grows, it is better to move to a package.

What we gain

  • The job no longer uses current: the environment always uses 3.13.15.1, the release it was created with, and the LIBPATH puts its lib first.
  • The PYBATCH packages are in its environment and do not touch the system Python or other users' Python.
  • The project records which Python version and which dependencies it needs.
  • The JCL runs a named command instead of the path of the script.
  • One project can provide several commands, such as spool-alert and spool-who.

References

© 2026 Emuframe·🍪 This site only uses functional cookies. No tracking or personal data collection.

On this page