A script to convert pyproject.toml
dependencies to environemnt.yaml
files.
The main goal of pyproject2conda
is to provide a means to keep all basic
dependency information, for both pip
based and conda
based environments, in
pyproject.toml
. I often use a mix of pip and conda when developing packages,
and in my everyday workflow. Some packages just aren't available on both. If you
use poetry, I'd highly recommend poetry2conda.
- Simple comment based syntax to add information to dependencies when creating
environment.yaml
This package is actively used by the author, but is still very much a work in progress. Please feel free to create a pull request for wanted features and suggestions!
Use one of the following to install pyproject2conda
:
$ pip install pyproject2conda
or
$ conda install -c conda-forge pyproject2conda
If using pip, to install with rich and shellingham support, either install them your self, or use:
$ pip install pyproject2conda[all]
The conda-forge distribution of typer (which pyproject2conda
uses) installs
rich and shellingham by default.
Consider the toml
file
test-pyproject.toml.
[project]
name = "hello"
requires-python = ">=3.8,<3.11"
dependencies = [
"athing", # p2c: -p # a comment
"bthing", # p2c: -s "bthing-conda"
"cthing; python_version < '3.10'", # p2c: -c conda-forge
]
# ...
Note the comment lines # p2c:...
. These are special tokens that
pyproject2conda
will analyze. The basic options are:
usage: -c [-h] [-c CHANNEL] [-p] [-s] [packages ...]
Parser searches for comments '# p2c: [OPTIONS] CONDA-PACKAGES
positional arguments:
packages
options:
-h, --help show this help message and exit
-c CHANNEL, --channel CHANNEL
Channel to add to the pyproject requirement
-p, --pip If specified, install pyproject dependency with pip
-s, --skip If specified skip pyproject dependency on this line
So, if we run the following, we get:
$ pyproject2conda yaml -f tests/data/test-pyproject.toml
channels:
- conda-forge
dependencies:
- bthing-conda
- conda-forge::cthing
- pip
- pip:
- athing
Note that other comments can be mixed in.
By default, the python version is not included in the resulting conda output. To
include the specification from pyproject.toml
, use --python-include infer
option:
$ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-include infer
channels:
- conda-forge
dependencies:
- python>=3.8,<3.11
- bthing-conda
- conda-forge::cthing
- pip
- pip:
- athing
While using comments to mark options has the convenience of placing the changes
right next to the dependency, it can becore a bit cumbersome. If you feel this
way, then you can use an alternative method to map pip
dependencies to conda
dependencies. For this, use the tool.pyproject2.conda.dependencies
table. For
example, we can do the same thing as above with:
# ...
[tool.pyproject2conda.dependencies]
athing = { pip = true }
bthing = { skip = true, packages = "bthing-conda" }
cthing = { channel = "conda-forge" }
pytest = { channel = "conda-forge" }
matplotlib = { skip = true, packages = [
"additional-thing; python_version < '3.9'",
"conda-matplotlib"
] }
build = { channel = "pip" }
# ...
To specify a specific value of python in the output, pass a value with:
$ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-include \
python=3.9
channels:
- conda-forge
dependencies:
- python=3.9
- bthing-conda
- conda-forge::cthing
- pip
- pip:
- athing
Note that this is for including python in the resulting environment file.
You can also constrain packages by the python version using the standard
pyproject.toml
syntax "...; python_version < 'some-version-number'"
. For is
parsed for both the pip packages and conda packages:
$ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-version 3.10
channels:
- conda-forge
dependencies:
- bthing-conda
- pip
- pip:
- athing
It is common to want to specify the python version and include it in the resulting environment file. You could, for example use:
$ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-version 3.10 \
--python-include python=3.10
channels:
- conda-forge
dependencies:
- python=3.10
- bthing-conda
- pip
- pip:
- athing
Because this is common, you can also just pass the option -p/--python
:
$ pyproject2conda yaml -f tests/data/test-pyproject.toml --python 3.10
channels:
- conda-forge
dependencies:
- python=3.10
- bthing-conda
- pip
- pip:
- athing
You can also add additional conda and pip dependencies with the flags
-d/--deps
and -r/--reqs
, respectively. Adding the last example:
$ pyproject2conda yaml -f tests/data/test-pyproject.toml -d dep -r req
channels:
- conda-forge
dependencies:
- bthing-conda
- conda-forge::cthing
- dep
- pip
- pip:
- athing
- req
These will also obey dependencies like dep:python_version<={version}
. Pass the
flags multiple times to pass multiple dependencies.
The name pyproject2conda
can be a bit long to type. For this reason, the
package also ships with the alias p2c
, which has the exact same functionality.
Additionally, the subcommands can be shortened to a unique match:
$ p2c y -f tests/data/test-pyproject.toml --python 3.10
channels:
- conda-forge
dependencies:
- python=3.10
- bthing-conda
- pip
- pip:
- athing
You can also call with python -m pyproject2conda
.
Given the extra dependency:
# ...
[project.optional-dependencies]
test = [
"pandas",
"pytest", # p2c: -c conda-forge
]
dev-extras = [
# p2c: -s "additional-thing; python_version < '3.9'" # additional pkg
## p2c: -s "another-thing" # skipped because of ## before p2c.
"matplotlib", # p2c: -s conda-matplotlib
]
dev = ["hello[test]", "hello[dev-extras]"]
dist-pypi = [
# this is intended to be parsed with --no-base option
"setuptools",
"build", # p2c: -p
]
# ...
and running the following gives:
$ pyproject2conda yaml -f tests/data/test-pyproject.toml -e test
channels:
- conda-forge
dependencies:
- bthing-conda
- conda-forge::cthing
- conda-forge::pytest
- pandas
- pip
- pip:
- athing
pyproject2conda
also works with self referenced dependencies:
$ pyproject2conda yaml -f tests/data/test-pyproject.toml -e dev
channels:
- conda-forge
dependencies:
- additional-thing
- bthing-conda
- conda-forge::cthing
- conda-forge::pytest
- conda-matplotlib
- pandas
- pip
- pip:
- athing
This also shows that p2c
comments without dependencies are also parsed. To
comment out such lines, make sure p2c
is preceded by ##
.
By default, pyproject2conda
includes a header in most output files to note
that the files are auto generated. No header is included by default when writing
to standard output. To override this behavior, pass --header/--noheader
:
$ pyproject2conda yaml -f tests/data/test-pyproject.toml --header
#
# This file is autogenerated by pyproject2conda
# with the following command:
#
# $ pyproject2conda yaml -f tests/data/test-pyproject.toml --header
#
# You should not manually edit this file.
# Instead edit the corresponding pyproject.toml file.
#
channels:
- conda-forge
dependencies:
- bthing-conda
- conda-forge::cthing
- pip
- pip:
- athing
pyproject2conda
can also be used within python:
>>> from pyproject2conda.requirements import ParseDepends
>>> p = ParseDepends.from_path("./tests/data/test-pyproject.toml")
# Basic environment
>>> print(p.to_conda_yaml(python_include="infer").strip())
channels:
- conda-forge
dependencies:
- python>=3.8,<3.11
- bthing-conda
- conda-forge::cthing
- pip
- pip:
- athing
# Environment with extras
>>> print(p.to_conda_yaml(extras="test").strip())
channels:
- conda-forge
dependencies:
- bthing-conda
- conda-forge::cthing
- conda-forge::pytest
- pandas
- pip
- pip:
- athing
pyproject2conda
can be configured with a [tool.pyproject2conda]
section in
pyproject.toml
. To specify conda channels use:
# ...
[tool.pyproject2conda]
channels = ['conda-forge']
# these are the same as the default values of `p2c project`
template_python = "py{py}-{env}"
template = "{env}"
style = "yaml"
# options
python = ["3.10"]
# Note that this is relative to the location of pyproject.toml
user_config = "config/userconfig.toml"
default_envs = ["test", "dev", "dist-pypi"]
[tool.pyproject2conda.envs.base]
style = ["requirements"]
# Note that the default value for `extras` is the name of the environment.
# To have no extras, either pass
# extras = []
# or
#
extras = false
#
# A value of `extras = true` also implies using the environment name
# as the extras.
[tool.pyproject2conda.envs."test-extras"]
extras = ["test"]
style = ["yaml", "requirements"]
[[tool.pyproject2conda.overrides]]
envs = ['test-extras', "dist-pypi"]
base = false
[[tool.pyproject2conda.overrides]]
envs = ["test", "test-extras"]
python = ["3.10", "3.11"]
Note that specifying channels at the command line overrides
tool.pyproject2conda.channels
.
You can also specify environments without the base dependencies (those under
project.dependencies
) by passing the --no-base
flag. This is useful for
defining environments for build, etc, that do not require the package be
installed. For example:
# ...
dist-pypi = [
# this is intended to be parsed with --no-base option
"setuptools",
"build", # p2c: -p
]
# ...
These can be accessed using either of the following:
$ pyproject2conda yaml -f tests/data/test-pyproject.toml -e dist-pypi --no-base
channels:
- conda-forge
dependencies:
- setuptools
- pip
- pip:
- build
or
>>> from pyproject2conda.requirements import ParseDepends
>>> p = ParseDepends.from_path("./tests/data/test-pyproject.toml")
# Basic environment
>>> print(p.to_conda_yaml(extras='dist-pypi', include_base=False).strip())
channels:
- conda-forge
dependencies:
- setuptools
- pip
- pip:
- build
pyproject2conda
provides a means to create all needed environment/requirement
files in one go. We configure the environments using the pyproject.toml
files
in the [tool.pyproject2conda]
section. For example, example the configuration:
# ...
[tool.pyproject2conda]
channels = ['conda-forge']
# these are the same as the default values of `p2c project`
template_python = "py{py}-{env}"
template = "{env}"
style = "yaml"
# options
python = ["3.10"]
# Note that this is relative to the location of pyproject.toml
user_config = "config/userconfig.toml"
default_envs = ["test", "dev", "dist-pypi"]
[tool.pyproject2conda.envs.base]
style = ["requirements"]
# Note that the default value for `extras` is the name of the environment.
# To have no extras, either pass
# extras = []
# or
#
extras = false
#
# A value of `extras = true` also implies using the environment name
# as the extras.
[tool.pyproject2conda.envs."test-extras"]
extras = ["test"]
style = ["yaml", "requirements"]
[[tool.pyproject2conda.overrides]]
envs = ['test-extras', "dist-pypi"]
base = false
[[tool.pyproject2conda.overrides]]
envs = ["test", "test-extras"]
python = ["3.10", "3.11"]
run through the command pyproject2conda project
(or p2c project
):
$ p2c project -f tests/data/test-pyproject.toml --dry
# --------------------
# Creating requirements base.txt
athing
bthing
cthing;python_version<"3.10"
# --------------------
# Creating yaml py310-test-extras.yaml
channels:
- conda-forge
dependencies:
- python=3.10
- conda-forge::pytest
- pandas
# --------------------
# Creating yaml py311-test-extras.yaml
channels:
- conda-forge
dependencies:
- python=3.11
- conda-forge::pytest
- pandas
# --------------------
# Creating requirements test-extras.txt
pandas
pytest
# --------------------
# Creating yaml py310-test.yaml
channels:
- conda-forge
dependencies:
- python=3.10
- bthing-conda
- conda-forge::pytest
- pandas
- pip
- pip:
- athing
# --------------------
# Creating yaml py311-test.yaml
channels:
- conda-forge
dependencies:
- python=3.11
- bthing-conda
- conda-forge::pytest
...
Note that here, we have used the --dry
option to just print the output. In
production, you'd omit this flag, and files according to --template
and
--template-python
would be used.
The options under [tool.pyproject2conda]
follow the command line options
(replace -
with _
). To specify an environment, you can either use the
[tool.pyproject.envs."environment-name"]
method, or, if the environment is the
same as the "extras" name, you can just specify it under
tool.pyproject2conda.default_envs
:
[tool.pyproject2conda]
# ...
default_envs = ["test"]
is equivalent to
[tool.pyproject2conda.envs.test]
extras = ["tests"]
To specify a conda environment (yaml
) file, pass style = "yaml"
(the
default). To specify a requirements file, pass style = "requirements"
. You can
specify both to make both.
Options in a given tool.pyproject2conda.envs."environemnt-name"
section
override those at the tool.pyproject2conda
level. So, for example:
# ...
[tool.pyproject2conda.envs."test-extras"]
extras = ["test"]
style = ["yaml", "requirements"]
# ...
will override use the two styles instead of the default of yaml
.
You can also override options for multiple environments using the
[[tools.pyproject2conda.overrides]]
list. Just specify the override option(s)
and the environments to apply them to. For example, above we specify that the
base option is False
for envs test-extras
and dist-pypi
, and that the
python version should be 3.10
and 3.11
for envs test
and test-extras
.
So in all, options are picked up, in order, from the environment definition, then the overrides list, and finally, from the default options.
You can also define "user defined" configurations. This can be done through the
option --user-config
. This allows you to define your own environments outside
of the (most likely source controlled) pyproject.toml
file. For example, we
have the option user_config=config/userconfig.toml
.
[tool.pyproject2conda.envs."user-dev"]
extras = ["dev", "dist-pypi"]
deps = ["extra-dep"]
reqs = ["extra-req"]
name = "hello"
Note that the full path of this file is note that the path of the user_conifg
file is relative to thempyproject.toml
file. So, if the pyproject.toml
file
is at a/path/pyproject.toml
, the path of user configuration files will be
a/path/config/userconfig.toml
. We then can run the following:
$ p2c project -f tests/data/test-pyproject.toml --dry --envs user-dev
# --------------------
# Creating yaml py310-user-dev.yaml
name: hello
channels:
- conda-forge
dependencies:
- python=3.10
- bthing-conda
- conda-forge::pytest
- conda-matplotlib
- extra-dep
- pandas
- setuptools
- pip
- pip:
- athing
- build
- extra-req
See command line interface documentation for details on the commands and options.
The application pyproject2conda
is used in the development of the following
packages:
cmomy
thermoextrap
tmmc-lnpy
module-utilities
analphipy
pyproject2conda
itself!
See the documentation for a look at pyproject2conda
in action.
This is free software. See LICENSE.
The author can be reached at wpk@nist.gov.
This package was created using Cookiecutter with the usnistgov/cookiecutter-nist-python template.