Examples#

This pages demonstrates Typed Setting’s features in a real application context. We’ll reimplement the settings loading functionality of various existing (and probably well-known) tools and libraries. In this way we combine the known with the new.

Black#

Black <https://github.com/psf/black>_ – the uncompromising code formatter - uses a project’s pyproject.toml for its settings. It finds the file even when you are in a sub directory of the project. Settings can be overridden by command line arguments.

What you’ll learn#

  • Using pyproject.toml and finding it from sub directories.

  • Using an Enum for settings values.

  • Adding command line arguments.

The settings file#

# pyproject.toml

[tool.black]
line-length = 79

The code#

First we define our settings class and the Enum that holds all supported Python versions.

We than create a click command. The decorator click_options() creates commnand line options and makes Click pass an instance of our settings class to the CLI function.

The decorator needs to know our settings class and which loaders to use.

We’ll use the default loaders and tell them:

  • how our app is called

  • which config files to load (find “pyproject.toml”)

  • the config file section to read from

  • to disable loading settings from environment varialbes

from enum import Enum

import click

import typed_settings as ts


class PyVersion(Enum):
    """
    Python versions that we support.
    """

    py37 = "3.7"
    py38 = "3.8"
    py39 = "3.9"


@ts.settings
class Settings:
    """
    Black settings.

    We limit ourselves to three options.
    """

    line_length: int = 88
    skip_string_normalization: bool = False
    target_version: PyVersion = PyVersion.py39  # Better: Auto-detect


@click.command()
@ts.click_options(
    Settings,
    ts.default_loaders(
        appname="black",
        config_files=[ts.find("pyproject.toml")],
        config_file_section="tool.black",
        env_prefix=None,
    ),
)
def cli(settings: Settings) -> None:
    print(settings)


if __name__ == "__main__":
    cli()

Trying it out#

Before we run our code, we’ll take a look at the generated --help. Note, that the default line length is 79, which we read from pyproject.toml.

$ python black.py --help
Usage: black.py [OPTIONS]

Options:
  --line-length INTEGER           [default: 79]
  --skip-string-normalization / --no-skip-string-normalization
                                  [default: no-skip-string-normalization]
  --target-version [py37|py38|py39]
                                  [default: py39]
  --help                          Show this message and exit.
$ python black.py --skip-string-normalization
Settings(line_length=79, skip_string_normalization=True, target_version=<PyVersion.py39: '3.9'>)

Python-Gitlab#

Python-gitlab is a GitLab API client. It loads configuration from a config file in your home directory. That file has multiple sections: One for general settings and one for each GitLab account that you use.

What you’ll learn#

  • Loading config files from a fixed location.

  • Loading different settings types from a single file.

  • Using secret settings.

The settings file#

# python-gitlab.toml

[global]
default = "gitlab-com"

[gitlab-com]
url = "https://gitlab.com"
private_token = "a93af93ff0adf9j3"
api_version = 4

[kde]
url = "https://invent.kde.org"
private_token = "0ad73a9e0341cee0"
api_version = 4

The code#

We need two different settings schemes: One for the global settings and one for the GitLab account settings. We declare our API tokens as secret settings This way, they won’t be shown in clear text when we print our settings.

We also need to call load() twice. Both calls use the same application name and load the same config files, but they read different sections from it.

Note

We load the the settings file from the current working directory in this example. In real life, you should use platformdirs as shown in the commented line.

# python_gitlab.py

import typed_settings as ts


@ts.settings
class GlobalSettings:
    default: str
    ssl_verify: bool = True


@ts.settings
class GitlabAccountSettings:
    url: str
    private_token: str = ts.secret()
    api_version: int = 3


appname = "python-gitlab"
# config_files = [platformdirs.user_config_path().joinpath(f"{appname}.toml")]
config_files = [f"{appname}.toml"]

global_settings = ts.load(
    GlobalSettings,
    appname=appname,
    config_files=config_files,
    config_file_section="global",
)
account_settings = ts.load(
    GitlabAccountSettings,
    appname=appname,
    config_files=config_files,
    config_file_section=global_settings.default,
)
print(global_settings)
print(account_settings)

Trying it out#

$ python python_gitlab.py
GlobalSettings(default='gitlab-com', ssl_verify=True)
GitlabAccountSettings(url='https://gitlab.com', private_token='*******', api_version=4)

As you can see, we loaded the [general] section and the [gitlab-com] section.

Also note that the API token is not displayed in clear text but as ***. This makes it safe to pass loaded settings to your favorite logger.

.pypirc#

The file ~/.pypirc is, for example, used by twine which publishes your Python packages to PyPI.

What you’ll learn#

  • Loading config files from a fixed location.

  • Loading different settings types from a single file.

  • Avoiding a [global] section for just listing other sections.

  • Using secret settings.

This example is implemented in two variants.

Original#

The original version uses a [global] section to list all configured accounts. A config file section must exist for each entry in that list.

It’s implementation is very similar to the Python-Gitlab example.

The settings file#

# pypirc.toml

[distutils]
index-servers = ["pypi", "test"]

[pypi]
repository = "https://upload.pypi.org/legacy/"
username = "test"

[test]
repository = "https://test.pypi.org/legacy/"
username = "test"

The code#

# pypirc_0.py

import sys
from typing import List

import typed_settings as ts


@ts.settings
class RepoServer:
    repository: str
    username: str
    password: str = ts.secret(default="")


@ts.settings
class Settings:
    index_servers: List[str]


settings = ts.load(Settings, "distutils", ["pypirc.toml"])
repos = {
    name: ts.load(RepoServer, name, ["pypirc.toml"])
    for name in settings.index_servers
}
REPO_NAME = sys.argv[1]
print(repos[REPO_NAME])

Trying it out#

$ python pypirc_0.py pypi
RepoServer(repository='https://upload.pypi.org/legacy/', username='test', password='')

Improved#

The second one uses nested settings classes and TOML’s support for dictionaries to avoid the [global] section.

The main difference is that Settings.repos is now a dict[str, Repository] instead of a list[str], so it’s a lot easier to load and acces the repo settings.

The settings file#

# pypirc.toml

[distutils.repos.pypi]
repository = "https://upload.pypi.org/legacy/"
username = "test"

[distutils.repos.test]
repository = "https://test.pypi.org/legacy/"
username = "test"

The code#

# pypirc_1.py

import sys
from typing import Dict

import typed_settings as ts


@ts.settings
class RepoServer:
    repository: str
    username: str
    password: str = ts.secret(default="")


@ts.settings
class Settings:
    repos: Dict[str, RepoServer]


settings = ts.load(Settings, "distutils", ["pypirc.toml"])
REPO_NAME = sys.argv[1]
print(settings.repos[REPO_NAME])

Trying it out#

$ python pypirc_1.py pypi
RepoServer(repository='https://upload.pypi.org/legacy/', username='test', password='')

Pytest#

Pytest is the testing framework for Python.

You can configure it with a config file and override each option via a command line flag.

Pytest is also dynamically extensible with plugins. Plugins can add new config options and command line arguments. To keep the --help output somewhat readable, all options are grouped by plugin.

What you’ll learn#

  • Dynamically creating settings for applications with a plug-in system.

  • Customizing how click options are create. This includes: - Setting help texts - Change the parameter declaration (“what the option looks like”) - Create on-way switches for boolean options

  • Creating command line applications for your settings

  • Using click options groups for nested settings

The code#

We first create two settings classes for our plugins: Coverage and :Emoji.

The option Coverage.src override its option name to be --cov instead of --prefix-src.

We then define Pytest’s base settings. They also override the Click parameter declarations and add help texts for the options.

The next step is to combine the base settings with the settings of of all Plugins. We use combine() for that which is a convenience wrapper around attrs.make_class().

Now that we habe class that contains Pytests own settings as well as the settings of all plugins, we can create a command line application. We create option groups by passing OptionGroupFactory to click_options().

from typing import Tuple

import click

import typed_settings as ts
from typed_settings.click_utils import OptionGroupFactory


@ts.settings
class Coverage:
    """
    Coverage settings
    """

    src: str = ts.option(default="", click={"param_decls": ("--cov", "src")})
    report: str = ""


@ts.settings
class Emoji:
    """
    Emoji settings
    """

    enable: bool = True


@ts.settings
class Base:
    """
    Main settings
    """

    marker: str = ts.option(
        default="",
        help="only run tests which macht the given substring expression",
        click={"param_decls": ("-m",)},
    )
    exitfirst: bool = ts.option(
        default=False,
        help="Exit instantly on first error or failed test",
        click={"param_decls": ("--exitfirst", "-x"), "is_flag": True},
    )
    stepwise: bool = ts.option(
        default=False,
        help=(
            "Exit on test failure and continue from last failing test next "
            "time"
        ),
        click={"param_decls": ("--stepwise", "--sw"), "is_flag": True},
    )


Settings = ts.combine(
    "Settings",
    Base,
    {
        # Imagine, this dict comes from a "load_plugins()" function :)
        "cov": Coverage(),
        "emoji": Emoji(),
    },
)


@click.command()
@click.argument("file_or_dir", nargs=-1)
@ts.click_options(Settings, "pytest", decorator_factory=OptionGroupFactory())
def cli(
    settings: Settings,  # type: ignore[valid-type]
    file_or_dir: Tuple[str, ...],
):
    print(settings)


if __name__ == "__main__":
    cli()

Trying it out#

$ python pytest.py --help
Usage: pytest.py [OPTIONS] [FILE_OR_DIR]...

Options:
  Main settings:
    -m TEXT                       only run tests which macht the given
                                  substring expression
    -x, --exitfirst               Exit instantly on first error or failed test
    --stepwise, --sw              Exit on test failure and continue from last
                                  failing test next time
  Coverage settings:
    --cov TEXT
    --cov-report TEXT
  Emoji settings:
    --emoji-enable / --no-emoji-enable
                                  [default: emoji-enable]
  --help                          Show this message and exit.
$ python pytest.py --sw --cov=src --emoji-enable
Settings(marker='', exitfirst=False, stepwise=True, cov=Coverage(src='src', report=''), emoji=Emoji(enable=True))

You can see that the help output contains all options from Pytest itself and all loaded plugins. Their options are also nicely grouped together and have help texts.

The second invocation prints the loaded settings. You can see how the combined settings class looks like.