Announcing Poetry 1.2.0

The Poetry team is pleased to announce the immediate availability of Poetry 1.2.0.

Poetry 1.2 boasts a massive list of changes, new features and fixes developed over the course of 2 years, with contributions from dozens of committers.

If you have a previous version of Poetry installed via the new installer, pipx or manually, getting Poetry 1.2.0 is as easy as:

$ poetry self update

$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3 - --uninstall

Since there are many changes in Poetry 1.2.0, this post details the changes over the following sections:

• Breaking changes and major features
• Other noteworthy changes and features
• Minor changes
• New commands
• Changes to existing commands
• New configuration options
• FAQ

For a complete list of changes, you can refer to the project history. Full documentation for Poetry 1.2 is available here. Any bugs or regressions should be reported to the issue tracker after checking for duplicates.

Breaking changes and major features #

New standalone installer #

The legacy get-poetry.py installation script has been replaced by install.python-poetry.org. The installer is now a standalone project, with its own issue tracker.

Most users will be satisfied by the defaults of the installer, which can be piped directly to a Python interpreter:

# Linux, macOS, Windows (WSL)
$ curl -sSL https://install.python-poetry.org | python3 -

# Windows (Powershell)
(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -

The new installer brings the following improvements:

• Releases are installed from standard wheels instead of special platform-specific archives. This allows for support of pipx and manual installs, and prevents a release being necessary to make use of newer dependency versions.
• Standard install locations are used for both Poetry itself, and the poetry CLI wrapper.
• Installations can be performed from a local path, or from a Git repository (including refs like branches or commits).
• Guidance on modifying $PATH is provided, but the user’s configuration is not altered by automated tools.
• Poetry will be installed using the Python interpreter the installer is invoked with.

The new installer (or a compatible install method such as pipx or manual installation) is additionally necessary for the new plugin system, detailed below.

If you wish to install using another method, or have advanced requirements like installing Poetry from git, refer to the full documentation.

Dropping support for managing Python 2.7 projects #

Python 2.7 has reached end of life over 18 months ago, on January 1, 2020.

Poetry 1.2 drops support for managing Python 2.7 projects, as:

• It led to increasing technical debt and slowed the development of Poetry.
• Projects have long had the time to migrate to Python 3.

Dropping support for Python 2.7, 3.5 and 3.6 as runtimes #

Poetry 1.2 drops runtime support for Python 2.7, 3.5 and 3.6. Running Poetry on these versions is now untested and unsupported.

Dependency groups #

Poetry provides a way to organize your dependencies by groups. For instance, you might have dependencies that are only needed to test your project or to build the documentation.

To declare a new dependency group, use a tool.poetry.group.<group> section where <group> is the name of your dependency group (for instance, test):

[tool.poetry.group.test]  # This section can be omitted

[tool.poetry.group.test.dependencies]
pytest = "^7.1.0"
pytest-mock = "*"

The dependencies declared in tool.poetry.dependencies are part of an implicit main group.

[tool.poetry.dependencies]  # The implicit `main` dependency group
httpx = "*"
pendulum = "*"

[tool.poetry.group.test.dependencies]
pytest = "^7.1.0"
pytest-mock = "*"

Dependency groups, other than the implicit main group, should generally contain additional dependencies that are part of your development process, as installing them is only possible using Poetry and poetry install.

If your project has additional dependencies meant to add additional functionality at runtime, they should be declared using the ecosystem-standard extras instead. Extras are supported by package build and install tools such as pip.

[tool.poetry.dev-dependencies]
pytest = "^7.1.0"
pytest-mock = "*"

[tool.poetry.group.dev.dependencies]
pytest = "^7.1.0"
pytest-mock = "*"

Optional groups #

A dependency group can be declared as optional. This makes sense when you have a group of dependencies that are only required in a specific environment or for a specialized purpose.

[tool.poetry.group.docs]
optional = true

[tool.poetry.group.docs.dependencies]
mkdocs = "*"

Optional groups can be installed in addition to the default dependencies by using the --with flag of the install command:

$ poetry install --with docs

Adding a dependency to a group #

The --group (-G) flag of the add command is the preferred way to add dependencies to a group:

$ poetry add pytest --group test

If the group does not already exist, it will be created automatically.

Installing group dependencies #

By default, dependencies across all non-optional groups will be installed when executing poetry install.

You can exclude one or more groups with the --without option:

$ poetry install --without docs,test

You can also opt in to optional groups by using the --with option:

$ poetry install --with docs

If you only want to install the default, non-grouped dependencies (aka the main group), you can do so with the --only option:

$ poetry install --only main

Finally, if you wish to install only specific groups of dependencies without installing the main group, the --only option can be used to do so:

$ poetry install --only docs,test

Removing dependencies from a group #

The remove command supports a --group (-G) flag to remove packages from a specific group:

$ poetry remove mkdocs --group docs

Plugin support #

Poetry now supports a experimental plugin system to alter or expand functionality.

Example use cases include functionality not desirable to the majority of Poetry users, features out of scope to the main Poetry project, or specialized functionality specific to a project.

The plugin system is designed to allow use of Poetry in these situations without requiring a custom fork.

Using plugins #

Poetry automatically loads all plugins installed into its environment.

While there are many methods to add plugins to a Python environment (or virtual environment), Poetry comes with a suite of self commands that should work regardless of install method:

$ poetry self add poetry-plugin-<NAME>

The self add command will ensure that the plugin is compatible with the current version of Poetry and install any necessary dependencies.

Any package specification understood by the standard add command is compatible with self add.

If you no longer need a plugin and want to uninstall it, you can use the self remove command:

$ poetry self remove poetry-plugin-<NAME>

You can also list all currently installed and discovered plugins by using self show plugins:

$ poetry self show plugins

Full documentation for installing and using plugins (including with other install methods) is available here.

Creating a plugin #

Early documentation for creating a plugin is available here.

Migration of the poetry export command #

The export command provides a way to export a list of locked dependencies to foreign formats, such as requirements.txt. This command was a feature added to make migration to Poetry easier, or to enable hybrid workflows, but it was never considered part of the core functionality of Poetry.

To reflect this, and to accelerate development of the main Poetry project, it has been migrated into a separate repo and is distributed separately as poetry-plugin-export. Note that it is now considered a separate project with its own issue tracker and release cycle.

To ease the transition, the plugin is installed by default for the 1.2 release. Future releases of Poetry will deprecate this automatic install, and require the user to explicitly install the plugin.

Other noteworthy changes and features #

Support for yanked releases (PEP 592) #

Poetry now supports yanked releases as defined by PEP 592, for both PyPI and any PEP 503-compatible repository.

Adding a dependency version that is yanked, or installing a project that depends on yanked releases will now raise a warning:

$ poetry add cryptography==37.0.3

[...]
Warning: The locked version 37.0.3 for cryptography is a yanked version. Reason for being yanked: Regression in OpenSSL.

$ poetry install

[...]
Warning: The file chosen for install of cryptography 37.0.3 (cryptography-37.0.3-cp36-abi3-manylinux_2_24_x86_64.whl) is yanked. Reason for being yanked: Regression in OpenSSL.

Support for Direct Origin URL records (PEP 610) #

Poetry now supports reading and writing PEP 610 records, which resolves edge cases and performance issues relating to determining the origin of installed dependencies.

Subdirectory support for Git dependencies #

It is now possible to specify a subdirectory from which Poetry should build and install a Git-based dependency.

The syntax used by the add command is the same as pip install/PEP 508 – a #subdirectory= fragment appended to the URL:

$ poetry add git+https://github.com/myorg/mypackage_with_subdirs.git#subdirectory=subdir

Manual editing of the pyproject.toml is supported as well. Full documentation, including examples is available here.

Single page repository support #

Poetry now supports discovering and installing dependencies from the ‘single page’ style of repository. Some widely-consumed package are not hosted in a PEP 503-compliant repository, but are instead listed on a single HTML page.

To add a single page repository as a source add it like any other repository:

$ poetry source add <SOURCE_NAME> <PAGE_URL>

# e.g.
$ poetry source add jax https://storage.googleapis.com/jax-releases/jax_releases.html

Full documentation is available here.

Synchronizing the environment with the lock file #

To ensure that the environment exactly matches the lock file, the install command has gained a new --sync flag:

$ poetry install --sync

The --sync option can be combined with any of the dependency group-related flags as expected:

$ poetry install --without dev --sync
$ poetry install --with docs --sync
$ poetry install --only dev

Please note that use of this command in the system environment (a common practice in containerized environments) may have unexpected results. --sync is intended only for use in a virtual environment where installed packages are exclusively managed by Poetry.

Opting out of binary distributions #

A new installer.no-binary setting has been introduced, to allow opting out of binary distributions of selected dependencies. This is functionally similar to pip install --no-binary.

This option can be configured globally to affect all usage of Poetry, but is best combined with --local to scope it to a specific project:

# Skip all binaries
$ poetry config --local installer.no-binary :all:
# Skip specific package binaries
$ poetry config --local installer.no-binary httpx,uvicorn
# Do not skip any binaries (default)
$ poetry config --local installer.no-binary :none:

Full documentation of this feature (including configuration using environment variables for CI or containers) is available here.

Native Python git client #

Poetry has robust (and improving) support for Git dependencies, which has always been enabled by the system git command. However, not all environments in which you want to use Poetry have a Git client available.

Poetry 1.2 introduces Git support based on Dulwich, a native Python implementation of the Git protocol, format, and client. Dulwich supports all operations Poetry requires, and should mostly be a drop-in replacement for Poetry’s previous usage of the git CLI.

However, as this is a major change, there is an escape hatch in the form of the experimental.system-git-client setting. When this is set to true, Poetry will revert to using your system’s git command.

This option may be necessary for users with lock files that contained an abbreviated Git commit sha, as the current Dulwich usage is unable to expand abbreviated hashes that are not directly pointed to by a ref (tag or branch). You can also simply update your lock file to use the unabbreviated form of the hash instead. For more information and discussion, see issue 6455.

Note that this option is experimental, and will be removed in a future release of Poetry.

Detection of the currently active Python (experimental) #

Due to refactoring required to enable plugin support and alternative install methods, Poetry lost the ability to detect the currently activated Python (aka the current python3 command in the $PATH), as selected by tools like pyenv or update-alternatives.

To ease the workflow of those using such tools, a new experimental virtualenvs.prefer-active-python setting has been introduced. If this is set to true, Poetry will attempt to detect the currently active Python interpreter when creating a new environment. This new method should function regardless of the install method used.

$ poetry config virtualenvs.prefer-active-python true
$ pyenv local 3.9.3
# The resulting environment should be created using Python 3.9.3
$ poetry install

Minor changes #

PEP 508 dependency specification parsing #

The add command now supports full PEP 508-style dependency specifications, enabling the addition of complex dependency definitions using ecosystem-standard syntax:

$ poetry add 'pytest-xdist[psutil] (>=2.4.0,<2.5.0); python_version >= "3.7"'

This command would result in the following addition to pyproject.toml:

[tool.poetry.dependencies]
pytest-xdist = {version = ">=2.4.0,<2.5.0", markers = "python_version >= \"3.7\"", extras = ["psutil"]}

Comprehensive HTTPS certificate support #

Poetry has long supported the use of custom certificates for repository access. However, not all code paths made use of these configured credentials, preventing some commands like poetry update from functioning properly against custom repos using certificate-based authentication.

Poetry 1.2 has significantly refactored both the repository access and HTTP request components, ensuring that certificates are uniformly applied to all relevant requests.

Non-verbose error handling #

Poetry 1.2 significantly reduces the verbosity of most common errors, by printing only the exception and not a partial stack trace:

$ poetry add httpx==0.0.0

Could not find a matching version of package httpx

For debugging and development work, different levels of verbosity are now available:

• --verbose (-v) to display the last stack frame (similar to Poetry 1.1)
• -vv to display a reduced stack trace, highlighting the exact error and the calls that led to it
• -vvv for maximum verbosity, printing a full stack trace and enabling debug logging

Management of setuptools and pip #

Poetry 1.2 will properly lock and manipulate the versions of setuptools, pip, and wheel in the target environment. This is to support projects which depend on pip, which up to this point could be managed by Poetry. However, this does lead to a sharp edge for users who use both Poetry 1.2 and 1.1.

As Poetry 1.1 will remove optional dependencies that are not requested, and as it considers setuptools, pip, and wheel to always be optional, they will be removed when a lock file that contains these packages is encountered. This is generally a rare edge case (and if you need to lock pip or setuptools you likely have fully migrated to 1.2), but some packages incorrectly declare a build-time dependency on pip or setuptools as a runtime dependency, and may trigger this edge case.

If setuptools, pip, or wheel are present in your lock file, you should fully migrate to 1.2, or at least lock with 1.1 until you can perform a full migration. For more information and discussion, see issue 4242.

Keyring backend issues #

Poetry 1.2’s support for keyring is nascent, and while extensive manual testing and CI have proven the base functionality, the diversity of possible backends and system configurations has proven to require more robust error handling and invocation in Poetry.

If Keyring is unable to automatically determine that the backend is locked/unavailable and fall back to the null backend, you can opt out yourself with the environment variable PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring.

We don’t expect this to become mandatory for environments in which the Keyring is not actively being used, but it serves as both a useful troubleshooting step and a good bailout for environments in which Keyring is not working, and you do not wish to troubleshoot Keyring so it can successfully communicate with your backend. For more information and discussion, see issue 1917.

Usefulness of experimental.new-installer false #

Due to bugs in older versions of pip, the parallel installer may sometimes experience race conditions. Modern pip versions should be highly reliable, but either due to edge cases in pip, Poetry, or simply having to use an older version, these race conditions can rarely occur.

When troubleshooting or working around them, please use the configuration setting installer.max-workers 1 where prior documentation and issues may have suggested experimental.new-installer false. The old installer is deprecated and has not received more than basic maintenance work, and may not interact perfectly with some new features in Poetry. Likewise, fixing any bugs in the new installer is preferred to depending on the old installer code, which is to be removed in the near future.

For more information and discussion, see issue 3336.

New commands #

self #

The self namespace groups subcommands related to management of Poetry and its runtime environment.

This namespace previously contained the [self update][self update docs] command, but has now been significantly expanded:

self add #

The self add command adds a dependency to Poetry’s runtime environment, similar to poetry add:

$ poetry self add poetry-plugin-<NAME>

self install #

The self install command ensures all configured packages are installed into Poetry’s runtime environment, similar to poetry install:

$ poetry self install
$ poetry self install --sync

This is useful when ~/.config/pypoetry/pyproject.toml is managed as part of a dotfiles repo or is mounted into a container. Note that the path of the runtime pyproject.toml may vary based on platform or if $POETRY_HOME is set.

self lock #

The self lock command ensures all configured packages are recorded to a runtime environment poetry.lock file, similar to poetry lock:

$ poetry self lock

self remove #

The self remove command removes a package from Poetry’s runtime environment, similar to poetry remove:

$ poetry self remove poetry-plugin-<NAME>

self show #

The self show command lists all configured runtime environment packages, similar to poetry show:

$ poetry self show

self show plugins #

The self show plugins command lists all discovered plugins in Poetry’s runtime environment:

$ poetry self show plugins

source #

The source namespace groups subcommands related to management of package sources (repositories).

Previously, configuring sources required manual edits to pyproject.toml.

source add #

The source add command adds a new source configuration to pyproject.toml:

$ poetry source add pypi-test https://test.pypi.org/simple/

source show #

The source show command displays information on all configured sources for the project:

$ poetry source show

Optionally, you can limit output to one or more sources by specifying them by name:

$ poetry source show pypi-test

source remove #

The source remove command removes a configured source from pyproject.toml:

$ poetry source remove pypi-test

Changes to existing commands #

Poetry 1.2 brings changes to several commands, primarily related to dependency groups.

Global options #

• Added --no-cache to disable all uses of cached packages and source contents

about #

• Now displays Poetry and poetry-core versions

add #

• Added --editable to add a dependency in editable mode
• Added --group / -G to add a dependency to a specific dependency group
• Now understands PEP 508-style dependency specifications

env remove #

• Added --all to delete all environments
• It is now possible to remove multiple environments in one invocation by specifying multiple names

install #

• Added --all-extras to install all extras
• Added --only to strictly install one or more dependency groups (ignoring the implicit main group)
• Added --only-root to install the project without any dependencies
• Added --with to select one or more dependency groups to install in addition to the implicit main group
• Added --without to skip installing one or more dependency groups
• Added --sync to ensure installed dependencies exactly match the lock file (and specified groups)
• Deprecated --no-dev (use --without dev instead)
• Deprecated --remove-untracked (use --sync instead)

lock #

• Added --check to verify poetry.lock agrees with pyproject.toml

new #

• Added --readme to specify the README file format

publish #

• Added --skip-existing to ignore errors from files already existing in the repository

run #

poetry run now parses arguments correctly, using the same logic as other Poetry commands. This means that the argument terminator -- is now consumed by poetry run instead of being passed through.

For example, poetry run tox -- arg1 arg2 would previously have been interpreted as ["tox", "--", "arg1", "arg2"]. It will now result in ["tox", "arg1", "arg2"] as the -- was interpreted as an argument to poetry run. If you need to express a -- in your command line, you will have to express it twice – once for Poetry, and once for the command being run, e.g. poetry run -- tox -- arg1 arg2. For more information and discussion, see issue 6440.

show #

• Added --only to strictly select one or more dependency groups to show (ignoring the implicit main group)
• Added --why to interrogate why a dependency is required (works standalone, or with --tree for a single package)
• Added --with to select one or multiple dependency groups to show in addition to the implicit main group
• Added --without to skip selecting one or more dependency groups
• Deprecated --no-dev (use --without dev instead)
• show <PACKAGE> will now list dependencies that require PACKAGE

shell #

• nushell is now supported
• Windows support has been improved (UX should be more similar to Unix platforms)

remove #

• Added --group / -G to remove a dependency from a specific dependency group

version #

• Added --dry-run to simulate a version bump without actually applying it

New configuration options #

experimental.system-git-client #

experimental.system-git-client causes Poetry to make use of the system git binary instead of Dulwich.

installer.max-workers #

installer.max-workers limits the maximum number of parallel workers used by the installer.

installer.no-binary #

installer.no-binary causes the installer to ignore some or all binary distributions, forcing installation from a source distribution.

virtualenvs.options.always-copy #

virtualenvs.options.always-copy causes the --always-copy flag to be passed to virtualenv during environment creation, ensuring that all necessary files are copied instead of linked.

virtualenvs.options.no-pip #

virtualenvs.options.no-pip causes the --no-pip flag to be passed to virtualenv during environment creation, preventing the automatic installation of pip into the environment.

virtualenvs.options.no-setuptools #

virtualenvs.options.no-setuptools causes the --no-setuptools flag to be passed to virtualenv during environment creation, preventing the automatic installation of setuptools into the environment.

virtualenvs.options.system-site-packages #

virtualenvs.options.system-site-packages causes the --system-site-packages flag to be passed to virtualenv during environment creation, allowing for the system site-packages directory (e.g. packages installed using the distro package manager) to be discoverable inside the virtual environment.

virtualenvs.prefer-active-python #

virtualenvs.prefer-active-python causes Poetry to attempt to detect the currently active python3 binary, and use it as the interpreter for creation of virtual environments. If false (the default), Poetry will instead use the same interpreter it was installed with.

virtualenvs.prompt #

virtualenvs.prompt allows for customization of the prompt used by Poetry-managed environments. Two template variables, {{project_name}} and {{python_version}} are available.

FAQ #

Are lock files compatible between 1.2 and 1.1? #

Yes, Poetry 1.2 will understand lock files generated by 1.1, and Poetry 1.1 will understand lock files generated by 1.2.

There will be inconsistencies related to specific formatting (e.g. case, order, indentation), so it is suggested to only commit lock files from one version.

If you notice any hard incompatibilities, please report them to the issue tracker.

Will 1.1 still be maintained? How long will 1.2 be maintained? #

Now that Poetry 1.2 is out, Poetry 1.1 is officially unmaintained.

1.2 will be maintained with fixes for major bugs and regressions until the release of the next stable version (1.3).

The Poetry team intends to significantly increase release cadence to prevent similarly long and painful release cycles. Users should expect to see new stable versions of Poetry released regularly.