Pyproject.toml support for DockerSettings (#3292)

* WIP pyproject toml support

* Docstrings

* Some docs

* Tests

* More tests

* Add link

* Fix install order mistake in docs

* Add support for installing local projects

* Add option to disable automatic requirements detection

* Fix some more conditions

* Local project installation docs

* Small docs and exception updates

* Remove wrong link

* Add enum to check

* Disable automatic detection by default

* Improve docs and log

* Adjust test to new default

Author: schustmi

docs/book/how-to/containerization/containerization.md

@@ -245,65 +245,102 @@ ZenML offers several ways to specify dependencies for your Docker containers:
 
 ### Python Dependencies
 
+By default, ZenML automatically installs all packages required by your active ZenML stack. 
+
+{% hint style="warning" %}
+In future versions, if none of the `replicate_local_python_environment`, `pyproject_path` or `requirements` attributes on `DockerSettings` are specified, ZenML will try to automatically find a `requirements.txt` and `pyproject.toml` files inside your current source root and install packages from the first one it finds. You can disable this behavior by setting `disable_automatic_requirements_detection=True`. If
+you already want this automatic detection in current versions of ZenML, set `disable_automatic_requirements_detection=False`.
+{% endhint %}
+
 1.  **Replicate Local Environment**:
+    ```python
+    docker_settings = DockerSettings(replicate_local_python_environment=True)
+
+
+    @pipeline(settings={"docker": docker_settings})
+    def my_pipeline(...):
+        ...
+    ```
+
+    This will run `pip freeze` to get a list of the installed packages in your local Python environment and will install them in the Docker image. This ensures that the same
+    exact dependencies will be installed.
+    {% hint style="warning" %}
+    This does not work when you have a local project installed. To install local projects, check out the `Install Local Projects` section below.
+    {% endhint %}
+2.  **Specify a `pyproject.toml` file**:
 
     ```python
-    # Use pip freeze (outputs a requirements file with exact package versions)
-    from zenml.config import DockerSettings, PythonEnvironmentExportMethod
-    docker_settings = DockerSettings(
-        replicate_local_python_environment=PythonEnvironmentExportMethod.PIP_FREEZE
-    )
-    # Or as a string
-    docker_settings = DockerSettings(replicate_local_python_environment="pip_freeze")
-
-    # Or use poetry (requires Poetry to be installed)
-    docker_settings = DockerSettings(
-        replicate_local_python_environment=PythonEnvironmentExportMethod.POETRY_EXPORT
-    )
-    # Or as a string
-    docker_settings = DockerSettings(replicate_local_python_environment="poetry_export")
-
-    # Use custom command (provide a list of command arguments)
-    docker_settings = DockerSettings(replicate_local_python_environment=[
-        "poetry", "export", "--extras=train", "--format=requirements.txt"
-    ])
+    docker_settings = DockerSettings(pyproject_path="/path/to/pyproject.toml")
+
+    @pipeline(settings={"docker": docker_settings})
+    def my_pipeline(...):
+        ...
     ```
 
-    This feature allows you to easily replicate your local Python environment in the Docker container, ensuring that your pipeline runs with the same dependencies.
-2.  **Specify Requirements Directly**:
+    By default, ZenML will try to export the dependencies specified in the `pyproject.toml` by trying to run `uv export` and `poetry export`.
+    If both of these commands do not work for your `pyproject.toml` file or you want to customize the command (for example to install certain
+    extras), you can specify a custom command using the `pyproject_export_command` attribute. This command must output a list of requirements following the format of the [requirements file](https://pip.pypa.io/en/stable/reference/requirements-file-format/). The command can contain a `{directory}` placeholder which will be replaced with the directory in which the `pyproject.toml` file is stored.
+
+    ```python
+    from zenml.config import DockerSettings
+
+    docker_settings = DockerSettings(pyproject_export_command=[
+        "uv",
+        "export",
+        "--extra=train",
+        "--format=requirements-txt"
+        "--directory={directory}
+    ])
+
+
+    @pipeline(settings={"docker": docker_settings})
+    def my_pipeline(...):
+        ...
+    ```
+3.  **Specify Requirements Directly**:
 
     ```python
     docker_settings = DockerSettings(requirements=["torch==1.12.0", "torchvision"])
     ```
-3.  **Use Requirements File**:
+4.  **Use Requirements File**:
 
     ```python
     docker_settings = DockerSettings(requirements="/path/to/requirements.txt")
     ```
-4.  **Specify ZenML Integrations**:
+5.  **Specify ZenML Integrations**:
 
     ```python
     from zenml.integrations.constants import PYTORCH, EVIDENTLY
 
     docker_settings = DockerSettings(required_integrations=[PYTORCH, EVIDENTLY])
     ```
-5.  **Control Stack Requirements**:\
+6.  **Control Stack Requirements**:
     By default, ZenML installs the requirements needed by your active stack. You can disable this behavior if needed:
 
     ```python
     docker_settings = DockerSettings(install_stack_requirements=False)
     ```
 
-{% hint style="info" %}
-You can combine these methods but do make sure that your list of requirements does not overlap with ones specified explicitly in the Docker settings to avoid version conflicts.
-{% endhint %}
+7.  **Install Local Projects**:
+    If your code requires the installation of some local code files as a python package, you can specify a command
+    that installs it as follows:
+    ```python
+    docker_settings = DockerSettings(local_project_install_command="pip install . --no-deps")
+    ```
+
+    {% hint style="warning" %}
+    Installing a local python package only works if your code files are included in the Docker image, so make sure you have
+    `allow_including_files_in_images=True` in your Docker settings. If you want to instead use the [code download functionality](#source-code-management)
+    to avoid building new Docker images for each pipeline run, you can follow [this example](https://github.com/zenml-io/zenml-patterns/tree/main/docker-local-pkg). 
+    {% endhint %}
 
 Depending on the options specified in your Docker settings, ZenML installs the requirements in the following order (each step optional):
 
 1. The packages installed in your local Python environment
 2. The packages required by the stack (unless disabled by setting `install_stack_requirements=False`)
 3. The packages specified via the `required_integrations`
-4. The packages specified via the `requirements` attribute
+4. The packages defined in the pyproject.toml file specified by the `pyproject_path` attribute
+5. The packages specified via the `requirements` attribute
 
 ### System Packages
 

src/zenml/config/build_configuration.py

@@ -135,6 +135,9 @@ def should_include_files(
         Returns:
             Whether files should be included in the image.
         """
+        if self.settings.local_project_install_command:
+            return True
+
         if self.should_download_files(code_repository=code_repository):
             return False
 
@@ -153,6 +156,9 @@ def should_download_files(
         Returns:
             Whether files should be downloaded in the image.
         """
+        if self.settings.local_project_install_command:
+            return False
+
         if self.should_download_files_from_code_repository(
             code_repository=code_repository
         ):
@@ -176,6 +182,9 @@ def should_download_files_from_code_repository(
         Returns:
             Whether files should be downloaded from the code repository.
         """
+        if self.settings.local_project_install_command:
+            return False
+
         if (
             code_repository
             and self.settings.allow_download_from_code_repository

src/zenml/config/docker_settings.py

@@ -91,12 +91,21 @@ class DockerSettings(BaseSettings):
     --------------------------------
     Depending on the configuration of this object, requirements will be
     installed in the following order (each step optional):
-    - The packages installed in your local python environment
+    - The packages installed in your local python environment (extracted using
+      `pip freeze`)
     - The packages required by the stack unless this is disabled by setting
-      `install_stack_requirements=False`.
+      `install_stack_requirements=False`
     - The packages specified via the `required_integrations`
+    - The packages defined inside a pyproject.toml file given by the
+      `pyproject_path` attribute.
     - The packages specified via the `requirements` attribute
 
+    If neither `replicate_local_python_environment`, `pyproject_path` or
+    `requirements` are specified, ZenML will try to automatically find a
+    requirements.txt or pyproject.toml file in your current source root
+    and installs packages from the first one it finds. You can disable this
+    behavior by setting `disable_automatic_requirements_detection=True`.
+
     Attributes:
         parent_image: Full name of the Docker image that should be
             used as the parent for the image that will be built. Defaults to
@@ -137,10 +146,29 @@ class DockerSettings(BaseSettings):
             packages.
         python_package_installer_args: Arguments to pass to the python package
             installer.
-        replicate_local_python_environment: If not `None`, ZenML will use the
-            specified method to generate a requirements file that replicates
-            the packages installed in the currently running python environment.
-            This requirements file will then be installed in the Docker image.
+        disable_automatic_requirements_detection: If set to True, ZenML will
+            not automatically detect requirements.txt files or pyproject.toml
+            files in your source root.
+        replicate_local_python_environment: If set to True, ZenML will run
+            `pip freeze` to gather the requirements of the local Python
+            environment and then install them in the Docker image.
+        pyproject_path: Path to a pyproject.toml file. If given, the
+            dependencies will be exported to a requirements.txt
+            formatted file using the `pyproject_export_command` and then
+            installed inside the Docker image.
+        pyproject_export_command: Command to export the dependencies inside a
+            pyproject.toml file to a requirements.txt formatted file. If not
+            given and ZenML needs to export the requirements anyway, `uv export`
+            and `poetry export` will be tried to see if one of them works. This
+            command can contain a `{directory}` placeholder which will be
+            replaced with the directory in which the pyproject.toml file is
+            stored.
+            **Note**: This command will be run before any code files are copied
+            into the image. It is therefore not possible to install a local
+            project using this command. This command should exclude any local
+            projects, and you can specify a `local_project_install_command`
+            instead which will be run after the code files are copied into the
+            image.
         requirements: Path to a requirements file or a list of required pip
             packages. During the image build, these requirements will be
             installed using pip. If you need to use a different tool to
@@ -149,13 +177,16 @@ class DockerSettings(BaseSettings):
         required_integrations: List of ZenML integrations that should be
             installed. All requirements for the specified integrations will
             be installed inside the Docker image.
-        required_hub_plugins: DEPRECATED/UNUSED.
         install_stack_requirements: If `True`, ZenML will automatically detect
             if components of your active stack are part of a ZenML integration
             and install the corresponding requirements and apt packages.
             If you set this to `False` or use custom components in your stack,
             you need to make sure these get installed by specifying them in
             the `requirements` and `apt_packages` attributes.
+        local_project_install_command: Command to install a local project in
+            the Docker image. This is run after the code files are copied into
+            the image, and it is therefore only possible when code is included
+            in the image, not downloaded at runtime.
         apt_packages: APT packages to install inside the Docker image.
         environment: Dictionary of environment variables to set inside the
             Docker image.
@@ -170,14 +201,6 @@ class DockerSettings(BaseSettings):
             from a code repository if possible.
         allow_download_from_artifact_store: If `True`, code can be downloaded
             from the artifact store.
-        build_options: DEPRECATED, use parent_image_build_config.build_options
-            instead.
-        dockerignore: DEPRECATED, use build_config.dockerignore instead.
-        copy_files: DEPRECATED/UNUSED.
-        copy_global_config: DEPRECATED/UNUSED.
-        source_files: DEPRECATED. Use allow_including_files_in_images,
-            allow_download_from_code_repository and
-            allow_download_from_artifact_store instead.
     """
 
     parent_image: Optional[str] = None
@@ -191,14 +214,18 @@ class DockerSettings(BaseSettings):
         PythonPackageInstaller.PIP
     )
     python_package_installer_args: Dict[str, Any] = {}
+    disable_automatic_requirements_detection: bool = True
     replicate_local_python_environment: Optional[
-        Union[List[str], PythonEnvironmentExportMethod]
+        Union[List[str], PythonEnvironmentExportMethod, bool]
     ] = Field(default=None, union_mode="left_to_right")
+    pyproject_path: Optional[str] = None
+    pyproject_export_command: Optional[List[str]] = None
     requirements: Union[None, str, List[str]] = Field(
         default=None, union_mode="left_to_right"
     )
     required_integrations: List[str] = []
     install_stack_requirements: bool = True
+    local_project_install_command: Optional[str] = None
     apt_packages: List[str] = []
     environment: Dict[str, Any] = {}
     user: Optional[str] = None
@@ -221,6 +248,8 @@ class DockerSettings(BaseSettings):
         "copy_global_config",
         "source_files",
         "required_hub_plugins",
+        "build_options",
+        "dockerignore",
     )
 
     @model_validator(mode="before")
@@ -305,6 +334,58 @@ def _validate_skip_build(self) -> "DockerSettings":
 
         return self
 
+    @model_validator(mode="after")
+    def _validate_code_files_included_if_installing_local_project(
+        self,
+    ) -> "DockerSettings":
+        """Ensures that files are included when installing a local package.
+
+        Raises:
+            ValueError: If files are not included in the Docker image
+                when trying to install a local package.
+
+        Returns:
+            The validated settings values.
+        """
+        if (
+            self.local_project_install_command
+            and not self.allow_including_files_in_images
+        ):
+            raise ValueError(
+                "Files must be included in the Docker image when trying to "
+                "install a local python package. You can do so by setting "
+                "the `allow_including_files_in_images` attribute of your "
+                "DockerSettings to `True`."
+            )
+
+        return self
+
+    @model_validator(mode="after")
+    def _deprecate_replicate_local_environment_commands(
+        self,
+    ) -> "DockerSettings":
+        """Deprecates some values for `replicate_local_python_environment`.
+
+        Returns:
+            The validated settings values.
+        """
+        if isinstance(
+            self.replicate_local_python_environment,
+            (str, list, PythonEnvironmentExportMethod),
+        ):
+            logger.warning(
+                "Specifying a command (`%s`) for "
+                "`DockerSettings.replicate_local_python_environment` is "
+                "deprecated. If you want to replicate your exact local "
+                "environment using `pip freeze`, set "
+                "`DockerSettings.replicate_local_python_environment=True`. "
+                "If you want to export requirements from a pyproject.toml "
+                "file, use `DockerSettings.pyproject_path` and "
+                "`DockerSettings.pyproject_export_command` instead."
+            )
+
+        return self
+
     model_config = ConfigDict(
         # public attributes are immutable
         frozen=True,

src/zenml/pipelines/build_utils.py

@@ -80,6 +80,11 @@ def requires_included_code(
     for step in deployment.step_configurations.values():
         docker_settings = step.config.docker_settings
 
+        if docker_settings.local_project_install_command:
+            # When installing a local package, we need to include the code
+            # files in the container image.
+            return True
+
         if docker_settings.allow_download_from_artifact_store:
             return False
 
@@ -136,6 +141,9 @@ def code_download_possible(
         Whether code download is possible for the deployment.
     """
     for step in deployment.step_configurations.values():
+        if step.config.docker_settings.local_project_install_command:
+            return False
+
         if step.config.docker_settings.allow_download_from_artifact_store:
             continue