add pip-compatible --group flag to uv pip install and uv pip compile (astral-sh#11686)

This is a minimal redux of astral-sh#10861 to be compatible with `uv pip`.

This implements the interface described in:
pypa/pip#13065 (comment) for `uv
pip install` and `uv pip compile`. Namely `--group <[path:]name>`, where
`path` when not defined defaults to `pyproject.toml`.

In that interface they add `--group` to `pip install`, `pip download`,
and `pip wheel`. Notably we do not define `uv pip download` and `uv pip
wheel`, so for parity we only need to implement `uv pip install`.
However, we also support `uv pip compile` which is not part of pip
itself, and `--group` makes sense there too.

----

The behaviour of `--group` for `uv pip` commands makes sense for the
cases upstream pip supports, but has confusing meanings in cases that
only we support (because reading pyproject.tomls is New Tech to them but
heavily supported by us). **Specifically case (h) below is a concerning
footgun, and case (e) below may get complaints from people who aren't
well-versed in dependency-groups-as-they-pertain-to-wheels.**


## Only Group Flags

Group flags on their own work reasonably and uncontroversially, except
perhaps that they don't do very clever automatic project discovery.

a) `uv pip install --group path/to/pyproject.toml:mygroup` pulls up
`path/to/project.toml` and installs all the packages listed by its
`mygroup` dependency-group (essentially treating it like another kind of
requirements.txt). In this regard it functions similarly to
`--only-group` in the rest of uv's interface.

b) `uv pip install --group mygroup` is just sugar for `uv pip install
--group pyproject.toml:mygroup` (**note that no project discovery
occurs**, upstream pip simply hardcodes the path "pyproject.toml" here
and we reproduce that.)

c) `uv pip install --group a/pyproject.toml:groupx --group
b/pyproject.toml:groupy`, and any other instance of multiple `--group`
flags, can be understood as completely independent requests for the
given groups at the given files.


## Groups With Named Packages

Groups being mixed with named packages also work in a fairly
unsurprising way, especially if you understand that things like
dependency-groups are not really supposed to exist on pypi, they're just
for local development.

d) `uv pip install mypackage --group path/to/pyproject.toml:mygroup`
much like multiple instances of `--group` the two requests here are
essentially completely independent: pleases install `mypackage`, and
please also install `path/to/pyproject.toml:mygroup`.

e) `uv pip install mypackage --group mygroup` is exactly the same, but
this is where it becomes possible for someone to be a little confused,
as you might think `mygroup` is supposed to refer to `mypackage` in some
way (it can't). But no, it's sourcing `pyproject.toml:mygroup` from the
current working directory.


## Groups With Requirements/Sourcetrees/Editables

Requirements and sourcetrees are where I expect users to get confused.
It behaves *exactly* the same as it does in the previous sections but
you would absolutely be forgiven for expecting a different behaviour.
*Especially* because `--group` with the rest of uv *does* do something
different.

f) `uv pip install -r a/pyproject.toml --group b/pyproject.toml:mygroup`
is again just two independent requests (install `a/pyproject.toml`'s
dependencies, and `b/pyproject.toml`'s `mygroup`).

g) `uv pip install -r pyproject.toml --group mygroup` is exactly like
the previous case but *incidentally* the two requests refer to the same
file. What the user wanted to happen is almost certainly happening, but
they are likely getting "lucky" here that they're requesting something
simple.

h) `uv pip install -r a/pyproject.toml --group mygroup` is again exactly
the same but the user is likely to get surprised and upset as this
invocation actually sources two different files (install
`a/pyproject.toml`'s dependencies, and `pyproject.toml`'s `mygroup`)! I
would expect most people to assume the `--group` flag here is covering
all applicable requirements/sourcetrees/editables, but no, it continues
to be a totally independent reference to a file with a hardcoded
relative path.

------

Fixes astral-sh#8590
Fixes astral-sh#8969

Author: Gankra

crates/uv-cli/src/lib.rs

@@ -15,7 +15,7 @@ use uv_configuration::{
     ProjectBuildBackend, TargetTriple, TrustedHost, TrustedPublishing, VersionControlSystem,
 };
 use uv_distribution_types::{Index, IndexUrl, Origin, PipExtraIndex, PipFindLinks, PipIndex};
-use uv_normalize::{ExtraName, GroupName, PackageName};
+use uv_normalize::{ExtraName, GroupName, PackageName, PipGroupName};
 use uv_pep508::{MarkerTree, Requirement};
 use uv_pypi_types::VerbatimParsedUrl;
 use uv_python::{PythonDownloads, PythonPreference, PythonVersion};
@@ -949,6 +949,7 @@ fn parse_maybe_string(input: &str) -> Result<Maybe<String>, String> {
 
 #[derive(Args)]
 #[allow(clippy::struct_excessive_bools)]
+#[command(group = clap::ArgGroup::new("sources").required(true).multiple(true))]
 pub struct PipCompileArgs {
     /// Include all packages listed in the given `requirements.in` files.
     ///
@@ -959,7 +960,7 @@ pub struct PipCompileArgs {
     ///
     /// The order of the requirements files and the requirements in them is used to determine
     /// priority during resolution.
-    #[arg(required(true), value_parser = parse_file_path)]
+    #[arg(group = "sources", value_parser = parse_file_path)]
     pub src_file: Vec<PathBuf>,
 
     /// Constrain versions using the given requirements files.
@@ -1022,6 +1023,14 @@ pub struct PipCompileArgs {
     #[arg(long, overrides_with("no_deps"), hide = true)]
     pub deps: bool,
 
+    /// Install the specified dependency group from a `pyproject.toml`.
+    ///
+    /// If no path is provided, the `pyproject.toml` in the working directory is used.
+    ///
+    /// May be provided multiple times.
+    #[arg(long, group = "sources")]
+    pub group: Vec<PipGroupName>,
+
     /// Write the compiled requirements to the given `requirements.txt` file.
     ///
     /// If the file already exists, the existing versions will be preferred when resolving
@@ -1586,6 +1595,14 @@ pub struct PipInstallArgs {
     #[arg(long, overrides_with("no_deps"), hide = true)]
     pub deps: bool,
 
+    /// Install the specified dependency group from a `pyproject.toml`.
+    ///
+    /// If no path is provided, the `pyproject.toml` in the working directory is used.
+    ///
+    /// May be provided multiple times.
+    #[arg(long, group = "sources")]
+    pub group: Vec<PipGroupName>,
+
     /// Require a matching hash for each requirement.
     ///
     /// By default, uv will verify any available hashes in the requirements file, but will not

crates/uv-distribution-types/src/annotation.rs

@@ -25,6 +25,9 @@ impl std::fmt::Display for SourceAnnotation {
                 RequirementOrigin::Project(path, project_name) => {
                     write!(f, "{project_name} ({})", path.portable_display())
                 }
+                RequirementOrigin::Group(path, project_name, group) => {
+                    write!(f, "{project_name} ({}:{group})", path.portable_display())
+                }
                 RequirementOrigin::Workspace => {
                     write!(f, "(workspace)")
                 }
@@ -40,6 +43,14 @@ impl std::fmt::Display for SourceAnnotation {
                     // Project is not used for override
                     write!(f, "--override {project_name} ({})", path.portable_display())
                 }
+                RequirementOrigin::Group(path, project_name, group) => {
+                    // Group is not used for override
+                    write!(
+                        f,
+                        "--override {project_name} ({}:{group})",
+                        path.portable_display()
+                    )
+                }
                 RequirementOrigin::Workspace => {
                     write!(f, "--override (workspace)")
                 }

crates/uv-normalize/src/group_name.rs

@@ -1,11 +1,15 @@
+use std::fmt::{Display, Formatter};
+use std::path::{Path, PathBuf};
 use std::str::FromStr;
 use std::sync::LazyLock;
 
 use serde::{Deserialize, Deserializer, Serialize, Serializer};
 
 use uv_small_str::SmallString;
 
-use crate::{validate_and_normalize_ref, InvalidNameError};
+use crate::{
+    validate_and_normalize_ref, InvalidNameError, InvalidPipGroupError, InvalidPipGroupPathError,
+};
 
 /// The normalized name of a dependency group.
 ///
@@ -82,6 +86,84 @@ impl AsRef<str> for GroupName {
     }
 }
 
+/// The pip-compatible variant of a [`GroupName`].
+///
+/// Either <groupname> or <path>:<groupname>.
+/// If <path> is omitted it defaults to "pyproject.toml".
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+#[cfg_attr(feature = "schemars", derive(schemars::JsonSchema))]
+pub struct PipGroupName {
+    pub path: Option<PathBuf>,
+    pub name: GroupName,
+}
+
+impl PipGroupName {
+    /// Gets the path to use, applying the default if it's missing
+    pub fn path(&self) -> &Path {
+        if let Some(path) = &self.path {
+            path
+        } else {
+            Path::new("pyproject.toml")
+        }
+    }
+}
+
+impl FromStr for PipGroupName {
+    type Err = InvalidPipGroupError;
+
+    fn from_str(path_and_name: &str) -> Result<Self, Self::Err> {
+        // The syntax is `<path>:<name>`.
+        //
+        // `:` isn't valid as part of a dependency-group name, but it can appear in a path.
+        // Therefore we look for the first `:` starting from the end to find the delimiter.
+        // If there is no `:` then there's no path and we use the default one.
+        if let Some((path, name)) = path_and_name.rsplit_once(':') {
+            // pip hard errors if the path does not end with pyproject.toml
+            if !path.ends_with("pyproject.toml") {
+                Err(InvalidPipGroupPathError(path.to_owned()))?;
+            }
+
+            let name = GroupName::from_str(name)?;
+            let path = Some(PathBuf::from(path));
+            Ok(Self { path, name })
+        } else {
+            let name = GroupName::from_str(path_and_name)?;
+            let path = None;
+            Ok(Self { path, name })
+        }
+    }
+}
+
+impl<'de> Deserialize<'de> for PipGroupName {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        let s = String::deserialize(deserializer)?;
+        Self::from_str(&s).map_err(serde::de::Error::custom)
+    }
+}
+
+impl Serialize for PipGroupName {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        let string = self.to_string();
+        string.serialize(serializer)
+    }
+}
+
+impl Display for PipGroupName {
+    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+        if let Some(path) = &self.path {
+            write!(f, "{}:{}", path.display(), self.name)
+        } else {
+            self.name.fmt(f)
+        }
+    }
+}
+
 /// The name of the global `dev-dependencies` group.
 ///
 /// Internally, we model dependency groups as a generic concept; but externally, we only expose the

crates/uv-normalize/src/lib.rs

@@ -3,7 +3,7 @@ use std::fmt::{Display, Formatter};
 
 pub use dist_info_name::DistInfoName;
 pub use extra_name::ExtraName;
-pub use group_name::{GroupName, DEV_DEPENDENCIES};
+pub use group_name::{GroupName, PipGroupName, DEV_DEPENDENCIES};
 pub use package_name::PackageName;
 
 use uv_small_str::SmallString;
@@ -121,6 +121,55 @@ impl Display for InvalidNameError {
 
 impl Error for InvalidNameError {}
 
+/// Path didn't end with `pyproject.toml`
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub struct InvalidPipGroupPathError(String);
+
+impl InvalidPipGroupPathError {
+    /// Returns the invalid path.
+    pub fn as_str(&self) -> &str {
+        &self.0
+    }
+}
+
+impl Display for InvalidPipGroupPathError {
+    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+        write!(
+            f,
+            "The `--group` path is required to end in 'pyproject.toml' for compatibility with pip; got: {}",
+            self.0,
+        )
+    }
+}
+impl Error for InvalidPipGroupPathError {}
+
+/// Possible errors from reading a [`PipGroupName`].
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub enum InvalidPipGroupError {
+    Name(InvalidNameError),
+    Path(InvalidPipGroupPathError),
+}
+
+impl Display for InvalidPipGroupError {
+    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+        match self {
+            InvalidPipGroupError::Name(e) => e.fmt(f),
+            InvalidPipGroupError::Path(e) => e.fmt(f),
+        }
+    }
+}
+impl Error for InvalidPipGroupError {}
+impl From<InvalidNameError> for InvalidPipGroupError {
+    fn from(value: InvalidNameError) -> Self {
+        Self::Name(value)
+    }
+}
+impl From<InvalidPipGroupPathError> for InvalidPipGroupError {
+    fn from(value: InvalidPipGroupPathError) -> Self {
+        Self::Path(value)
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;

crates/uv-pep508/src/origin.rs

@@ -1,6 +1,6 @@
 use std::path::{Path, PathBuf};
 
-use uv_normalize::PackageName;
+use uv_normalize::{GroupName, PackageName};
 
 /// The origin of a dependency, e.g., a `-r requirements.txt` file.
 #[derive(
@@ -12,6 +12,8 @@ pub enum RequirementOrigin {
     File(PathBuf),
     /// The requirement was provided via a local project (e.g., a `pyproject.toml` file).
     Project(PathBuf, PackageName),
+    /// The requirement was provided via a local project (e.g., a `pyproject.toml` file).
+    Group(PathBuf, PackageName, GroupName),
     /// The requirement was provided via a workspace.
     Workspace,
 }
@@ -22,6 +24,7 @@ impl RequirementOrigin {
         match self {
             RequirementOrigin::File(path) => path.as_path(),
             RequirementOrigin::Project(path, _) => path.as_path(),
+            RequirementOrigin::Group(path, _, _) => path.as_path(),
             // Multiple toml are merged and difficult to track files where Requirement is defined. Returns a dummy path instead.
             RequirementOrigin::Workspace => Path::new("(workspace)"),
         }

crates/uv-requirements/src/source_tree.rs

@@ -1,6 +1,6 @@
-use std::borrow::Cow;
-use std::path::Path;
+use std::path::{Path, PathBuf};
 use std::sync::Arc;
+use std::{borrow::Cow, collections::BTreeMap};
 
 use anyhow::{Context, Result};
 use futures::stream::FuturesOrdered;
@@ -18,7 +18,7 @@ use uv_pep508::RequirementOrigin;
 use uv_pypi_types::Requirement;
 use uv_resolver::{InMemoryIndex, MetadataResponse};
 use uv_types::{BuildContext, HashStrategy};
-use uv_warnings::warn_user_once;
+
 #[derive(Debug, Clone)]
 pub struct SourceTreeResolution {
     /// The requirements sourced from the source trees.
@@ -37,7 +37,7 @@ pub struct SourceTreeResolver<'a, Context: BuildContext> {
     /// The extras to include when resolving requirements.
     extras: &'a ExtrasSpecification,
     /// The groups to include when resolving requirements.
-    groups: &'a DependencyGroups,
+    groups: &'a BTreeMap<PathBuf, DependencyGroups>,
     /// The hash policy to enforce.
     hasher: &'a HashStrategy,
     /// The in-memory index for resolving dependencies.
@@ -50,7 +50,7 @@ impl<'a, Context: BuildContext> SourceTreeResolver<'a, Context> {
     /// Instantiate a new [`SourceTreeResolver`] for a given set of `source_trees`.
     pub fn new(
         extras: &'a ExtrasSpecification,
-        groups: &'a DependencyGroups,
+        groups: &'a BTreeMap<PathBuf, DependencyGroups>,
         hasher: &'a HashStrategy,
         index: &'a InMemoryIndex,
         database: DistributionDatabase<'a, Context>,
@@ -100,9 +100,13 @@ impl<'a, Context: BuildContext> SourceTreeResolver<'a, Context> {
 
         let mut requirements = Vec::new();
 
+        // Resolve any groups associated with this path
+        let default_groups = DependencyGroups::default();
+        let groups = self.groups.get(path).unwrap_or(&default_groups);
+
         // Flatten any transitive extras and include dependencies
         // (unless something like --only-group was passed)
-        if self.groups.prod() {
+        if groups.prod() {
             requirements.extend(
                 FlatRequiresDist::from_requirements(metadata.requires_dist, &metadata.name)
                     .into_iter()
@@ -116,20 +120,24 @@ impl<'a, Context: BuildContext> SourceTreeResolver<'a, Context> {
 
         // Apply dependency-groups
         for (group_name, group) in &metadata.dependency_groups {
-            if self.groups.contains(group_name) {
-                requirements.extend(group.iter().cloned());
+            if groups.contains(group_name) {
+                requirements.extend(group.iter().cloned().map(|group| Requirement {
+                    origin: Some(RequirementOrigin::Group(
+                        path.to_path_buf(),
+                        metadata.name.clone(),
+                        group_name.clone(),
+                    )),
+                    ..group
+                }));
             }
         }
         // Complain if dependency groups are named that don't appear.
-        // This is only a warning because *technically* we support passing in
-        // multiple pyproject.tomls, but at this level of abstraction we can't see them all,
-        // so hard erroring on "no pyproject.toml mentions this" is a bit difficult.
-        for name in self.groups.explicit_names() {
+        for name in groups.explicit_names() {
             if !metadata.dependency_groups.contains_key(name) {
-                warn_user_once!(
-                    "The dependency-group '{name}' is not defined in {}",
-                    path.display()
-                );
+                return Err(anyhow::anyhow!(
+                    "The dependency group '{name}' was not found in the project: {}",
+                    path.user_display()
+                ));
             }
         }