gh-134160: Start "Extending and embedding" with a Diataxis-style tutorial #142314

encukou · 2025-12-05T16:52:23Z

The "Extending and embedding" section of the docs starts with a "tutorial", which is now outdated (it uses soft-deprecated API), but it also doesn't quite work as a tutorial (in the Diátaxis sense).

This PR pulls out the bits needed for a simple extension module: it's as simple as it can get to expose a C function. Topics that need lengthy explanation are left out (this includes crucial ones like refcounting & error handling, put also modern things like ABI info or free-threading support).
The code is updated to modern, non-deprecated API -- specifically, PEP 793's PyModExport.

The remainder of the existing chapter is renamed to "Using the C API: Assorted topics", to mirror the later chapter “Defining Extension Types: Assorted Topics”. This title was somewhat fitting even without the tutorial part taken out.
Care is taken to not remove any information, unless it's duplicated or no longer relevant. An “assorted topics” section works nicely here.

It would be nice to pull more bits out into dedicated explanation or tutorial pages; that's out of scope for this PR.

I apologize for any typos; I found reviewers are much better at finding them than I am (especially after I've been rewriting drafts for days).

Issue: Docs Examples Focus on Outdated Single-Phase Init Modules and Static Types #134160

📚 Documentation preview 📚: https://cpython-previews--142314.org.readthedocs.build/

See: https://diataxis.fr/tutorials/#ruthlessly-minimise-explanation

Doc/extending/first-extension-module.rst

merwok · 2025-12-05T17:33:19Z

Doc/extending/index.rst

+   first-extension-module.rst
+   extending.rst
+   newtypes_tutorial.rst
+   newtypes.rst
+   building.rst
+   windows.rst
+   embedding.rst


Suggested change

first-extension-module.rst

extending.rst

newtypes_tutorial.rst

newtypes.rst

building.rst

windows.rst

embedding.rst

first-extension-module

extending

newtypes_tutorial

newtypes

building

windows

embedding

Why change this? All of CPython's toctrees have the .rst suffixes.

It’s not required, so is a little cleaner (like we don’t have to do import './module.py'), but maybe Sphinx didn’t support that when it was first created and now that’s the style we have.

Oh. Makes sense!
But I think the first such change should be its own PR, so it's easily revertable if anything goes wrong. This PR is big enough on its own.

Doc/extending/first-extension-module.rst

Co-authored-by: Stan Ulbrych <[email protected]> Co-authored-by: Éric <[email protected]>

encukou · 2025-12-10T11:47:44Z

I've changed things so the first compilation is after adding #includes. Not finding Python.h headers is a reasonable failure mode, and it makes sense to debug that together with any other tool issues.

encukou · 2025-12-12T15:30:28Z

The “now run your build tool” section stood out as the most no-tutorial-i part of the tutorial, so I tried the tools from the PyPA recommendation list and chose the most straightforward one: meson-python.
(It'd be setuptools if it weren't for pypa/distutils#387)

I put notes on using other tools in an appendix.

ngoldbaum · 2025-12-12T18:28:34Z

As a big proponent of meson-python IMO it's a great choice for this.

merwok · 2025-12-13T14:10:15Z

Doc/extending/first-extension-module.rst

+.. tip::
+
+   If you don't have ``pip`` installed, run ``python -m ensurepip``,
+   preferably in a :mod:`virtual environment <venv>`.


Do people actually need to run ensurepip if they have run python -m venv .venv ?

This tutorial could show the venv commands directly (and avoid ensurepip step) instead of referring to them.

I added “You need to be able to install Python packages” to the intro, leaving exact steps for setting that up out of scope here.
Does that work?

merwok · 2025-12-13T14:11:01Z

Doc/extending/first-extension-module.rst

+   preferably in a :mod:`virtual environment <venv>`.
+   You can also use another tool that can build and install
+   ``pyproject.toml``-based projects, like
+   `uv <https://docs.astral.sh/uv/>`_ (``uv pip install .``).


This note seems to contradict the intent of not providing options in a tutorial 🙂

This is partly setup you're expected to have before starting, and part troubleshooting advice.

I added it to the introduction, and made this note smaller. Does that work?

Will check!

ZeroIntensity

Yay, thanks for doing this! I'm excited to finally see a better tutorial.

I tried to avoid being overly nitpicky in my review.

Doc/extending/extending.rst

Doc/extending/first-extension-module.rst

Doc/extending/index.rst

Doc/extending/first-extension-module.rst

Co-authored-by: Daniele Nicolodi <[email protected]> Co-authored-by: Peter Bierma <[email protected]>

Doc/includes/capi-extension/spammodule-01.c

ZeroIntensity

Mostly just typos left. I left a few minor wording suggestions, so feel free to reject those if you want to.

ZeroIntensity · 2025-12-17T17:01:36Z

Doc/extending/first-extension-module.rst

+
+We will focus on giving you a "feel" of what Python's C API is like.
+It will not teach you important concepts, like error handling
+and reference counting, which are covered in later chapters.


This isn't currently incorrect, but I feel that this reads better:

Suggested change

and reference counting, which are covered in later chapters.

and reference counting, as those are covered in later chapters.

+1 for which (simpler, direct)

ZeroIntensity · 2025-12-17T17:02:21Z

Doc/extending/first-extension-module.rst

+On other systems, you might need to adjust some details -- for example,
+a system command name.
+
+You need to have suitable C compiler and Python development headers installed.


Suggested change

You need to have suitable C compiler and Python development headers installed.

You need to have a suitable C compiler and Python development headers installed.

ZeroIntensity · 2025-12-17T17:04:58Z

Doc/extending/first-extension-module.rst

+======================
+
+Begin by creating a directory for this tutorial, and switching to it
+on command line.


Suggested change

on command line.

on the command line.

Or:

Suggested change

on command line.

on your command line.

+1 for the

ZeroIntensity · 2025-12-17T17:06:42Z

Doc/extending/first-extension-module.rst

+   See `meson-python documentation <meson-python>`_ for details on
+   configuration.
+
+Now, build install the *project in the current directory* (``.``) via ``pip``:


Why is this italicized?

ZeroIntensity · 2025-12-17T17:08:20Z

Doc/extending/first-extension-module.rst

+extension.
+Unlike Python, C has an explicit compilation step.


These two sentences feel very related.

Suggested change

extension.

Unlike Python, C has an explicit compilation step.

extension, because unlike Python, C has an explicit compilation step.

Juxtaposition is a valid construct! 🙂

ZeroIntensity · 2025-12-17T17:14:06Z

Doc/extending/first-extension-module.rst

+   >>> print(spam)
+   <module 'spam' from '/home/encukou/dev/cpython/spam.so'>
+
+You have a extension module!


Suggested change

You have a extension module!

You have an extension module!

ZeroIntensity · 2025-12-17T17:16:24Z

Doc/extending/first-extension-module.rst

+One of the simplest ways to write glue code is a ":c:data:`METH_O`" function,
+which takes two Python objects and returns one.
+All Python objects -- regardless of the Python type -- are represented in C
+as pointers to the ``PyObject`` structure.


It would be nice to have a link here, since this is the first time we introduce it.

Suggested change

as pointers to the ``PyObject`` structure.

as pointers to the :c:type:`PyObject` structure.

ZeroIntensity · 2025-12-17T17:18:02Z

Doc/extending/first-extension-module.rst

+As with module slots, a zero-filled sentinel marks the end of the array.
+
+Next, we'll add the method to the module.
+Add a :c:data:`Py_mod_methods` slot to your a :c:type:`PyMethodDef` array:


Suggested change

Add a :c:data:`Py_mod_methods` slot to your a :c:type:`PyMethodDef` array:

Add a :c:data:`Py_mod_methods` slot to your :c:type:`PyMethodDef` array:

ZeroIntensity · 2025-12-17T17:18:55Z

Doc/extending/first-extension-module.rst

+but let's start with a fixed value, say, ``3``.
+
+The Python C API provides a function to create a Python :py:type:`int` object
+from a C ``int`` values: :c:func:`PyLong_FromLong`. [#why-pylongfromlong]_


Suggested change

from a C ``int`` values: :c:func:`PyLong_FromLong`. [#why-pylongfromlong]_

from a C ``int`` value: :c:func:`PyLong_FromLong`. [#why-pylongfromlong]_

ZeroIntensity · 2025-12-17T17:54:02Z

Doc/extending/first-extension-module.rst

+   }
+
+That's it for the setup.
+Now, all that is left is calling C library function :c:func:`system` with


Suggested change

Now, all that is left is calling C library function :c:func:`system` with

Now, all that is left is calling the C library function :c:func:`system` with

encukou added 7 commits December 5, 2025 17:29

New first chapter of a C API tutorial

4d31b9a

Reword more

4c2b982

Start from the beginning

1525795

TMP

dc844dd

Full draft

32ae620

Ruthlessly miminise explanation

13d5efa

See: https://diataxis.fr/tutorials/#ruthlessly-minimise-explanation

Re-add some reference tragets

7f1014e

encukou added the docs Documentation in the Doc dir label Dec 5, 2025

github-project-automation bot added this to Docs PRs Dec 5, 2025

github-project-automation bot moved this to Todo in Docs PRs Dec 5, 2025

bedevere-app bot added the awaiting core review label Dec 5, 2025

bedevere-app bot mentioned this pull request Dec 5, 2025

Docs Examples Focus on Outdated Single-Phase Init Modules and Static Types #134160

Open

Typo fix

4c61bf4

merwok reviewed Dec 5, 2025

View reviewed changes

Doc/extending/first-extension-module.rst Outdated Show resolved Hide resolved

merwok reviewed Dec 5, 2025

View reviewed changes

Doc/extending/first-extension-module.rst Outdated Show resolved Hide resolved

merwok reviewed Dec 5, 2025

View reviewed changes

Doc/extending/first-extension-module.rst Outdated Show resolved Hide resolved

merwok reviewed Dec 5, 2025

View reviewed changes

Doc/extending/first-extension-module.rst Show resolved Hide resolved

merwok reviewed Dec 5, 2025

View reviewed changes

Doc/extending/first-extension-module.rst Show resolved Hide resolved

merwok reviewed Dec 5, 2025

View reviewed changes

Doc/extending/first-extension-module.rst Outdated Show resolved Hide resolved

merwok reviewed Dec 5, 2025

View reviewed changes

Doc/extending/first-extension-module.rst Outdated Show resolved Hide resolved

StanFromIreland reviewed Dec 5, 2025

View reviewed changes

Doc/extending/first-extension-module.rst Outdated Show resolved Hide resolved

encukou and others added 4 commits December 10, 2025 11:32

Apply suggestions from code review

0d91185

Co-authored-by: Stan Ulbrych <[email protected]> Co-authored-by: Éric <[email protected]>

Address more review comments

2902a8b

tart with the headers

276d8cd

str->bytes is *encoding*

2306fcd

encukou added 3 commits December 12, 2025 14:03

Merge in the main branch

a312862

Link to the list of recommended tools in the Python Packaging User Guide

ae85969

Fix line highlight

27af529

Move signature check up

7069412

encukou added the skip news label Dec 12, 2025

encukou added 2 commits December 12, 2025 16:22

Use meson-python

54e2999

Use 'sampleproject' for the project name

f92562c

merwok reviewed Dec 13, 2025

View reviewed changes

ZeroIntensity reviewed Dec 15, 2025

View reviewed changes

dnicolodi reviewed Dec 16, 2025

View reviewed changes

Doc/extending/first-extension-module.rst Outdated Show resolved Hide resolved

encukou and others added 4 commits December 16, 2025 14:18

Apply suggestions from code review

a17458a

Co-authored-by: Daniele Nicolodi <[email protected]> Co-authored-by: Peter Bierma <[email protected]>

Address more review

638d2a0

Use :c:func:system, not system

2fc1511

Expand the explanation of Py_RETURN_NONE

24c6eaa

ngoldbaum reviewed Dec 16, 2025

View reviewed changes

Doc/includes/capi-extension/spammodule-01.c Outdated Show resolved Hide resolved

encukou added 2 commits December 17, 2025 11:37

Merge in the main branch

8e622ac

Add a comment for <stdlib.h>

772152c

ZeroIntensity reviewed Dec 17, 2025

View reviewed changes

	and reference counting, which are covered in later chapters.
	and reference counting, as those are covered in later chapters.

	You need to have suitable C compiler and Python development headers installed.
	You need to have a suitable C compiler and Python development headers installed.

		extension.
		Unlike Python, C has an explicit compilation step.

	extension.
	Unlike Python, C has an explicit compilation step.
	extension, because unlike Python, C has an explicit compilation step.

	as pointers to the ``PyObject`` structure.
	as pointers to the :c:type:`PyObject` structure.

	Add a :c:data:`Py_mod_methods` slot to your a :c:type:`PyMethodDef` array:
	Add a :c:data:`Py_mod_methods` slot to your :c:type:`PyMethodDef` array:

	from a C ``int`` values: :c:func:`PyLong_FromLong`. [#why-pylongfromlong]_
	from a C ``int`` value: :c:func:`PyLong_FromLong`. [#why-pylongfromlong]_

	Now, all that is left is calling C library function :c:func:`system` with
	Now, all that is left is calling the C library function :c:func:`system` with

Uh oh!

gh-134160: Start "Extending and embedding" with a Diataxis-style tutorial #142314

Are you sure you want to change the base?

gh-134160: Start "Extending and embedding" with a Diataxis-style tutorial #142314

Conversation

encukou commented Dec 5, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

encukou commented Dec 10, 2025

Uh oh!

encukou commented Dec 12, 2025

Uh oh!

ngoldbaum commented Dec 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZeroIntensity left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZeroIntensity left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

encukou commented Dec 5, 2025 •

edited by github-actions bot

Loading