Generating API Documentation for Your Python Package with Sphinx-apidoc¶

Introduction¶

In this guide, we will explore how to generate API documentation for your Python package using Sphinx-apidoc. Effective documentation is crucial for understanding, maintaining, and collaborating on software projects. Sphinx-apidoc is a tool that automates the process of creating documentation from docstrings within your Python codebase. Let's dive into the steps required to set up and utilize this tool effectively.

Motivation¶

In my quest to streamline the documentation process for my python projects as packages, I embarked on a journey to explore methods of leveraging docstrings to automatically generate documentation compatible with ReadTheDocs or MkDocs, platforms I'm familiar with and prefer due to their formatting styles. Despite my inclination towards MkDocs, I encountered a dilemma as it lacks built-in support for API documentation.

Amidst my research, I stumbled upon a pivotal discovery. A question posed by a fellow developer seeking to automatically generate Sphinx documentation from their docstrings led me to a solution. Users had proposed a script, which eventually found its way into the Sphinx ecosystem as sphinx-apidoc, aligning perfectly with my requirements.

Despite the prevalence of Sphinx in platforms like ReadTheDocs, I encountered discussions debating the merits of MkDocs versus Sphinx, particularly concerning module documentation. While MkDocs appealed to me for its simplicity, the absence of built-in support for API documentation posed a significant hurdle. This realization prompted me to explore alternatives and delve later on into lesser-known solutions like mkdocstrings or react based ones or whatever solution is used in pydandic api documentation, built using mkdocs with some automation.

With my decision made, I opted to utilize sphinx-apidoc for its efficiency and compatibility with Sphinx. My journey involved generating a foundational Sphinx documentation, employing sphinx-apidoc to automate the creation of documentation directives, integrating them into my project's index file, and finally, compiling the documentation. Alongside these steps, I undertook additional refinements, such as adopting the widely-used sphinx-rtd-theme for an enhanced visual experience and leveraging napoleon for greater flexibility in docstring formatting, eliminating the need for reStructuredText within the docstrings themselves.

Installing Sphinx¶

Before we can generate documentation with Sphinx-apidoc, we need to install Sphinx. Sphinx is a documentation generation tool widely used in the Python community.

pip install -U sphinx

sudo apt update
sudo apt install python3-sphinx

pip uninstall sphinx

sudo apt remove python3-sphinx
sudo apt autoremove

Generating Documentation Base Files for a Basic Setup¶

To start generating documentation with Sphinx-apidoc, we first need to create the base files for our documentation project. This can be achieved using the sphinx-quickstart command:

cd my/project/path
sphinx-quickstart docs

Follow the prompts to set up your project. Once completed, you can generate the HTML documentation by running:

cd docs
make html

Your documentation will be located at build/html/index.html.

Utilizing Docstrings¶

Incorporating Directives¶

Now that we have our basic documentation structure set up, let's incorporate our Python docstrings into the documentation. We can achieve this with Sphinx-apidoc. For example, if our module is located at ./statanalysis, we can generate documentation files for it using the following command:

sphinx-apidoc ./statanalysis/ -o ./docs/source/statanalysis/ -f -E

Make sure to update your index file like below to include the newly generated files.

...

.. toctree::
   :maxdepth: 2

   statanalysis/statanalysis

...

Here, i use statanalysis/statanalysis that will access the file docs/source/statanalysis/statanalysis.rts. This file list the other module files.

Then run the command make html in the docs directory to update the build.

Enhancing Docstring Readability¶

We can improve the readability of our docstrings by using the napoleon extension. napoleon allows us to write docstrings in a more concise and readable format, such as Google or Numpy style. To enable napoleon, update your conf.py with the following:

...
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
    "sphinx.ext.napoleon",
]

#  `napoleon` settings
# from https://github.com/cimarieta/sphinx-autodoc-example
napoleon_google_docstring = True
napoleon_numpy_docstring = True
napoleon_include_init_with_doc = True
napoleon_include_private_with_doc = True
napoleon_include_special_with_doc = True
napoleon_use_admonition_for_examples = False
napoleon_use_admonition_for_notes = False
napoleon_use_admonition_for_references = False
napoleon_use_ivar = False
napoleon_use_param = True
napoleon_use_rtype = True
...

Rerun the make html. Great! They may still be some warnings, but in general, the build will be successful.

Additional Extensions for Enhanced Functionality¶

Sphinx offers a variety of extensions to enhance the functionality of your documentation. Some useful extensions include:

• sphinx.ext.autodoc: Automatically generates documentation from docstrings.
• sphinx.ext.autosummary: Generates summaries of modules, classes, and functions based on documentation.
• sphinx.ext.mathjax: Renders mathematical equations in your documentation.
• sphinx.ext.todo: Includes "todo" items in your documentation.
• and more.

You can activate these extensions by adding them to the extensions list in your conf.py like below:

extensions = [
    "sphinx.ext.autodoc", 
    "sphinx.ext.autosummary",
    "sphinx.ext.napoleon",
    "sphinx.ext.mathjax",
    "sphinx.ext.todo",
]

extensions = [
   # like open3d:0.17
   "sphinx.ext.autodoc", # generate documentation for Python modules, classes, and functions, from docstring
   "sphinx.ext.autosummary", # generates summaries of modules, classes, and functions based on the documentation generated by sphinx.ext.autodoc
   "sphinx.ext.napoleon", # the  `napoleon` style of docstrings for Python modules, classes, and functions, is easy to read and write
   "sphinx.ext.mathjax", # display mathematical equations
   "sphinx.ext.todo", # include "todo" items in your documentation
   # "nbsphinx", #include Jupyter Notebooks in your documentation
   # 'm2r2', #convert Markdown files to reStructuredText (RST) format,
   # more from https://github.com/cimarieta/sphinx-autodoc-example
   # "nbsphinx", #include Jupyter Notebooks in your documentation
   # 'm2r2', #convert Markdown files to reStructuredText (RST) format,
   # more from https://github.com/cimarieta/sphinx-autodoc-example
   # 'sphinx.ext.doctest', # test code snippets in the documentation
   # 'sphinx.ext.intersphinx', # link to external documentation
   # 'sphinx.ext.coverage', # measure the coverage of the documentation
   # 'sphinx.ext.ifconfig', # include content based on configuration options
   # 'sphinx.ext.viewcode', # show the source code of modules and functions
   # 'sphinx.ext.githubpages', # publish documentation on GitHub Pages
]

Handy Tips for Integration¶

Author: Susan Li source: `LogisticRegressionImplementation.ipynb <https://github.com/aihubprojects/Logistic-Regression-From-Scratch-Python/blob/master/LogisticRegressionImplementation.ipynb>`_

def my_function():
 """
 This is a function that does something.

 .. todo:: Implement error handling.
 """

poetry add sphinx --group buildthedocs

pip install sphinx

Customizing the Theme¶

The default theme or even the built in themes (alabaster, classic, sphinxdoc, scrools, ...) provided by Sphinx may not suit your preferences.

You can customize the theme to better fit your project's style. Some popular themes include:

• solar-theme
• sphinx-rtd-theme
• furo

Install your chosen theme using pip and update the html_theme in your conf.py.

poetry add sphinx-rtd-theme --group buildthedocs

...
html_theme = 'sphinx_rtd_theme'
...

Here was my output of my python package statanalysis with sphinx-docapi + sphinx-rtd-theme

Integrating README.md into Sphinx Documentation¶

To include your README.md file as an introduction to your Sphinx documentation, you can use the m2r2 extension as implicitely mentioned in this stackoverflow answer. Follow these steps to seamlessly integrate your README.md file into your Sphinx documentation:

1. Install m2r2 Extension: First, install the m2r2 extension in your development environment using Poetry or pip:

   using poetry using pip

   poetry add m2r2 --group buildthedocs
   

   pip install m2r2
   

2. Create readme.rst File: Create a readme.rst file in your Sphinx documentation source directory (./docs/source/readme.rst).

   readme.rst using poetry

   In your readme.rst file, use the mdinclude directive to include the content of your README.md file:

   -----------
   Introduction
   -----------
   
   .. mdinclude:: ../../README.md
   

   Update your index.rst file to include the readme.rst file in the table of contents (toctree directive):

   ...
   
   .. toctree::
       :maxdepth: 2
       :caption: Contents:
   
       readme  # Include readme.rst here
       reference
   
   ...
   

By following these steps, you can seamlessly integrate your README.md file into your Sphinx documentation, providing an informative introduction to your project.

Deploying Documentation to ReadTheDocs¶

Once you've generated your Sphinx documentation, deploying it to ReadTheDocs allows you to host and share your documentation with ease.

To deploy your documentation to ReadTheDocs, you need to prepare your documentation and configuration files, then you can import your project to ReadTheDocs and initiate the build process. Follow this step-by-step guide:

1. Export Dependencies:

   Before deploying your documentation, you need to export your project dependencies to ensure that ReadTheDocs builds your documentation with the correct dependencies.

   Depending on your package manager (e.g., Poetry, pip), export your dependencies to a requirements.txt file.

   Depending on your package manager, to get the right syntax, refer to this cheat on package managers

   Example Using Poetry

   poetry export -f requirements.txt --output ./docs/requirements.txt --without-hashes --with buildthedocs
   

   This command exports the dependencies specified in your Poetry project, excluding development dependencies, and includes the dependencies grouped under buildthedocs.

2. Create Configuration File: Next, you need to create a .readthedocs.yaml file in the root directory of your project. This file specifies the build configuration for ReadTheDocs.

   Example of .readthedocs.yaml

   # Required
   version: 2
   
   # Set the OS, Python version, and other tools you might need
   build:
     os: ubuntu-22.04
     tools:
       python: "3.8"
   
   # Build documentation in the "docs/" directory with Sphinx
   sphinx:
     configuration: docs/source/conf.py
   
   # Optionally build your docs in additional formats such as PDF and ePub
   formats:
     - pdf
     - epub
   
   python:
     install:
       - requirements: docs/requirements.txt
   

   This configuration file specifies the Python version, operating system, and build tools required for building your documentation. Additionally, it defines the location of your Sphinx configuration file and specifies the dependencies needed for the build.

3. Create ReadTheDocs Account: If you haven't already, create an account on the ReadTheDocs website.

4. Import Project: Import your project to ReadTheDocs, linking it to your GitHub repository or other version control system. You can either use the GitHub integration or import your project manually.

   Manual Import Example

   • Navigate to the manual import page.

     

   • Enter the URL of your repository (e.g., https://github.com/hermann-web/lissajou)

   • Specify the desired project name.

     That name will appear in https://<name>.readthedocs.io/en/latest

   • Follow the prompts to complete the import process.

5. Build Documentation: Once your project is imported, initiate the build process on ReadTheDocs. The platform will automatically detect changes in your repository and trigger builds accordingly.

6. Monitor Builds on ReadTheDocs Dashboard: After initiating the build process, you can monitor the progress and status of your documentation builds on the ReadTheDocs dashboard. This allows you to track build successes, view logs, and troubleshoot any errors that may occur during the build process.

   

By following these steps, you can successfully deploy your Sphinx documentation to ReadTheDocs and make it accessible to your audience.

This was the steps for one python package i've built

# 1. add deps
poetry add sphinx==^7.0.0 --group buildthedocs
poetry add sphinx-rtd-theme --group buildthedocs
poetry add m2r2 --group buildthedocs
# 2. create the docu template
sphinx-quickstart docs
# 2. generate buildthedocs requirements, generate api doc and build docu
source scripts/run-build-docu.sh
# 3. add readme.rst
# 4. update index.rst to integrate readme.rst and openconv/modules.rst in 
# 5. update conf.py to specify main path, add extensions, modify the html_theme
# 6. rebuild docu to check
source scripts/run-build-docu.sh
# 7. add a .readthedocs file
# 8. push to readthedocs.org

Related Posts¶

• How to publish your python project (to pypi or testPypi) with Poetry then install as dependency in another project ?
• Cheat on Python Package Managers

Additional Resources¶

For more information on Sphinx and its capabilities, refer to the official Sphinx documentation. Additionally, you can explore the available themes and extensions to further enhance your documentation.

• Sphinx Documentation - PDF
• sphinx-autodoc-example - github
• Sphinx Quickstart
• lissajou Module Documentation
• statanalysis Module Documentation
• Readthedocs Configuration File Reference