Compiling with Setuptools#

Historically, setuptools and its predecessor distutils, were the main ways of building a package that uses Cython modules. Because of this, Cython ships with some convenience libraries to help use it with setuptools.

Declarative syntax#

Since late 2024 setuptools has supported a declarative syntax for building extension modules. In this one does not ship a setup.py file but instead describe your extension modules in pyproject.toml. This is simpler than the more commonly used setup.py version but somewhat more limited in how you can customize it.

Most of the Cython documentation pre-dates this system (and thus is described in terms of setup.py) but we include an example here for illustrative purposes:

# pyproject.toml
[build-system]
requires = ["setuptools", "cython"]
build-backend = "setuptools.build_meta"

[project]
name = "cython-example"  # as it would appear on PyPI
version = "0.1"

[tool.setuptools]
ext-modules = [
    {name = "cython_example.example", sources = ["example.pyx"]}
]

Basic `setup.py`#

The setuptools extension provided with Cython allows you to pass .pyx files directly to the Extension constructor in your setup file.

If you have a single Cython file that you want to turn into a compiled extension, say with filename example.pyx the associated setup.py would be:

setup.py#

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("example.pyx")
)

If your build depends directly on Cython in this way, then you may also want to inform pip that Cython is required for setup.py to execute, following PEP 518, creating a pyproject.toml file containing, at least:

pyproject.toml#

[build-system]
requires = ["setuptools", "Cython"]

To understand the setup.py more fully look at the official setuptools documentation. To compile the extension for use in the current directory use:

$ python setup.py build_ext --inplace

Note

setuptools 74.1.0 adds experimental support for extensions in pyproject.toml (instead of setup.py):

pyproject.toml#

[build-system]
requires = ["setuptools", "cython"]
build-backend = "setuptools.build_meta"

[project]
name = "mylib-foo"
version = "0.42"

[tool.setuptools]
ext-modules = [
  {name = "example", sources = ["example.pyx"]}
]

In this case, you can use any build frontend - e.g. build

$ python -m build

Configuring the C-Build#

Note

More details on building Cython modules that use cimport numpy can be found in the Numpy section of the user guide.

If you have Cython include files or Cython definition files in non-standard places you can pass an include_path parameter to cythonize:

from setuptools import setup
from Cython.Build import cythonize

setup(
    name="My hello app",
    ext_modules=cythonize("src/*.pyx", include_path=[...]),
)

If you need to specify compiler options, libraries to link with or other linker options you will need to create Extension instances manually (note that glob syntax can still be used to specify multiple extensions in one line):

from setuptools import Extension, setup
from Cython.Build import cythonize

extensions = [
    Extension("primes", ["primes.pyx"],
        include_dirs=[...],
        libraries=[...],
        library_dirs=[...]),
    # Everything but primes.pyx is included here.
    Extension("*", ["*.pyx"],
        include_dirs=[...],
        libraries=[...],
        library_dirs=[...]),
]
setup(
    name="My hello app",
    ext_modules=cythonize(extensions),
)

Some useful options to know about are

include_dirs- list of directories to search for C/C++ header files (in Unix form for portability),
libraries - list of library names (not filenames or paths) to link against,
library_dirs - list of directories to search for C/C++ libraries at link time.

Note that when using setuptools, you should import it before Cython, otherwise, both might disagree about the class to use here.

Often, Python packages that offer a C-level API provide a way to find the necessary C header files:

from setuptools import Extension, setup
from Cython.Build import cythonize

extensions = [
    Extension("*", ["*.pyx"],
        include_dirs=["/usr/local/include"]),
]
setup(
    name="My hello app",
    ext_modules=cythonize(extensions),
)

If your options are static (for example you do not need to call a tool like pkg-config to determine them) you can also provide them directly in your .pyx or .pxd source file using a special comment block at the start of the file:

# distutils: libraries = spam eggs
# distutils: include_dirs = /opt/food/include

If you cimport multiple .pxd files defining libraries, then Cython merges the list of libraries, so this works as expected (similarly with other options, like include_dirs above).

If you have some C files that have been wrapped with Cython and you want to compile them into your extension, you can define the setuptools sources parameter:

# distutils: sources = [helper.c, another_helper.c]

Note that these sources are added to the list of sources of the current extension module. Spelling this out in the setup.py file looks as follows:

from setuptools import Extension, setup
from Cython.Build import cythonize

sourcefiles = ['example.pyx', 'helper.c', 'another_helper.c']

extensions = [Extension("example", sourcefiles)]

setup(
    ext_modules=cythonize(extensions)
)

The Extension class takes many options, and a fuller explanation can be found in the setuptools documentation.

Sometimes this is not enough and you need finer customization of the setuptools Extension. To do this, you can provide a custom function create_extension to create the final Extension object after Cython has processed the sources, dependencies and # distutils directives but before the file is actually Cythonized. This function takes 2 arguments template and kwds, where template is the Extension object given as input to Cython and kwds is a dict with all keywords which should be used to create the Extension. The function create_extension must return a 2-tuple (extension, metadata), where extension is the created Extension and metadata is metadata which will be written as JSON at the top of the generated C files. This metadata is only used for debugging purposes, so you can put whatever you want in there (as long as it can be converted to JSON). The default function (defined in Cython.Build.Dependencies) is:

def default_create_extension(template, kwds):
    if 'depends' in kwds:
        include_dirs = kwds.get('include_dirs', []) + ["."]
        depends = resolve_depends(kwds['depends'], include_dirs)
        kwds['depends'] = sorted(set(depends + template.depends))

    t = template.__class__
    ext = t(**kwds)
    if hasattr(template, "py_limited_api"):
        ext.py_limited_api = template.py_limited_api
    metadata = dict(distutils=kwds, module_name=kwds['name'])
    return ext, metadata

In case that you pass a string instead of an Extension to cythonize(), the template will be an Extension without sources. For example, if you do cythonize("*.pyx"), the template will be Extension(name="*.pyx", sources=[]).

Just as an example, this adds mylib as library to every extension:

from Cython.Build.Dependencies import default_create_extension

def my_create_extension(template, kwds):
    libs = kwds.get('libraries', []) + ["mylib"]
    kwds['libraries'] = libs
    return default_create_extension(template, kwds)

ext_modules = cythonize(..., create_extension=my_create_extension)

Note

If you Cythonize in parallel (using the nthreads argument), then the argument to create_extension must be pickleable. In particular, it cannot be a lambda function.

Cythonize arguments#

The function cythonize() can take extra arguments which will allow you to customize your build.

Cython.Build.cythonize(module_list, exclude=None, nthreads=0, aliases=None, quiet=False, force=None, language=None, exclude_failures=False, show_all_warnings=False, **options)#

Compile a set of source modules into C/C++ files and return a list of distutils Extension objects for them.

Parameters:

module_list – As module list, pass either a glob pattern, a list of glob patterns or a list of Extension objects. The latter allows you to configure the extensions separately through the normal distutils options. You can also pass Extension objects that have glob patterns as their sources. Then, cythonize will resolve the pattern and create a copy of the Extension for every matching file.
exclude – When passing glob patterns as module_list, you can exclude certain module names explicitly by passing them into the exclude option.
nthreads – The number of concurrent builds for parallel compilation (requires the multiprocessing module).
aliases – If you want to use compiler directives like # distutils: ... but can only know at compile time (when running the setup.py) which values to use, you can use aliases and pass a dictionary mapping those aliases to Python strings when calling cythonize(). As an example, say you want to use the compiler directive # distutils: include_dirs = ../static_libs/include/ but this path isn’t always fixed and you want to find it when running the setup.py. You can then do # distutils: include_dirs = MY_HEADERS, find the value of MY_HEADERS in the setup.py, put it in a python variable called foo as a string, and then call cythonize(..., aliases={'MY_HEADERS': foo}).
quiet – If True, Cython won’t print error, warning, or status messages during the compilation.
force – Forces the recompilation of the Cython modules, even if the timestamps don’t indicate that a recompilation is necessary.
language – To globally enable C++ mode, you can pass language='c++'. Otherwise, this will be determined at a per-file level based on compiler directives. This affects only modules found based on file names. Extension instances passed into cythonize() will not be changed. It is recommended to rather use the compiler directive # distutils: language = c++ than this option.
exclude_failures – For a broad ‘try to compile’ mode that ignores compilation failures and simply excludes the failed extensions, pass exclude_failures=True. Note that this only really makes sense for compiling .py files which can also be used without compilation.
show_all_warnings – By default, not all Cython warnings are printed. Set to true to show all warnings.
annotate – If True, will produce a HTML file for each of the .pyx or .py files compiled. The HTML file gives an indication of how much Python interaction there is in each of the source code lines, compared to plain C code. It also allows you to see the C/C++ code generated for each line of Cython code. This report is invaluable when optimizing a function for speed, and for determining when to release the GIL: in general, a nogil block may contain only “white” code. See examples in Determining where to add types or Primes.
annotate-fullc – If True will produce a colorized HTML version of the source which includes entire generated C/C++-code.
compiler_directives – Allow to set compiler directives in the setup.py like this: compiler_directives={'embedsignature': True}. See Compiler directives.
depfile – produce depfiles for the sources if True.
cache – If True the cache enabled with default path. If the value is a path to a directory, then the directory is used to cache generated .c/.cpp files. By default cache is disabled. See Cython cache.

Multiple Cython Files in a Package#

To automatically compile multiple Cython files without listing all of them explicitly, you can use glob patterns:

setup(
    ext_modules = cythonize("package/*.pyx")
)

You can also use glob patterns in Extension objects if you pass them through cythonize():

extensions = [Extension("*", ["*.pyx"])]

setup(
    ext_modules = cythonize(extensions)
)

Distributing Cython modules#

Following recent improvements in the distribution toolchain, it is not recommended to include generated files in source distributions. Instead, require Cython at build-time to generate the C/C++ files, as defined in PEP 518 and PEP 621. See Basic setup.py.

It is, however, possible to distribute the generated .c files together with your Cython sources, so that users can install your module without needing to have Cython available.

Doing so allows you to make Cython compilation optional in the version you distribute. Even if the user has Cython installed, they may not want to use it just to install your module. Also, the installed version may not be the same one you used, and may not compile your sources correctly.

This simply means that the setup.py file that you ship with will just be a normal setuptools file on the generated .c files, for the basic example we would have instead:

from setuptools import Extension, setup

setup(
    ext_modules = [Extension("example", ["example.c"])]
)

This is easy to combine with cythonize() by changing the file extension of the extension module sources:

from setuptools import Extension, setup

USE_CYTHON = ...   # command line option, try-import, ...

ext = '.pyx' if USE_CYTHON else '.c'

extensions = [Extension("example", ["example"+ext])]

if USE_CYTHON:
    from Cython.Build import cythonize
    extensions = cythonize(extensions)

setup(
    ext_modules = extensions
)

If you have many extensions and want to avoid the additional complexity in the declarations, you can declare them with their normal Cython sources and then call the following function instead of cythonize() to adapt the sources list in the Extensions when not using Cython:

import os.path

def no_cythonize(extensions, **_ignore):
    for extension in extensions:
        sources = []
        for sfile in extension.sources:
            path, ext = os.path.splitext(sfile)
            if ext in ('.pyx', '.py'):
                if extension.language == 'c++':
                    ext = '.cpp'
                else:
                    ext = '.c'
                sfile = path + ext
            sources.append(sfile)
        extension.sources[:] = sources
    return extensions

If you want to expose the C-level interface of your library for other libraries to cimport from, use package_data to install the .pxd files, e.g.:

setup(
    package_data = {
        'my_package': ['*.pxd'],
        'my_package/sub_package': ['*.pxd'],
    },
    ...
)

These .pxd files need not have corresponding .pyx modules if they contain purely declarations of external libraries.

Compiling with Setuptools#

Declarative syntax#

Basic setup.py#

Configuring the C-Build#

Cythonize arguments#

Multiple Cython Files in a Package#

Distributing Cython modules#

Basic `setup.py`#