Installation of apache-beam[gcp] in a clean environment gets stuck in pip's dependency resolver.

[NicAlis]

Installing apache-beam[gcp]==2.34.0 or newer version into a clean virtual environment with pip 21.3 or newer currently gets stuck in pip's dependency resolver.

Workarounds

1. Preinstall following Apache Beam dependencies prior to installing Beam itself:

pip install google-api-core==1.31.6
pip install google-cloud-pubsub==2.13.1 google-cloud-bigquery-storage==2.13.2
pip install apache-beam[gcp]

2. Use apache-beam[gcp]==2.33.0 or earlier that are not as adversely affected.

---

Symptoms

One of the symptoms that shows numerous backtracking iterations performed by pip's dependency resolver are warnings like

 google-api-core 2.8.2 does not provide the extra 'grpcgcp

[juanan-hernandez]

+1 here

[richardcann]

As a temporary fix Ive had to use poetry instead of pip and it works fine. Would like to know why the dep tree for this using pip forces the installation of google-api-core 2.8.2 which I can see has removed the extra grpcgcp in this pull request of latest version

[Abacn]

This is due to an upstream change of google-cloud-bigquery-storage which bumped google-cloud-core version requirements. This is fixed in #22205 . You can also apply this change to your docker file used.

[Abacn]

The 'grpcgcp' warning is popped when installing google-cloud-spanner v1.x, which used this extra for google-api-core that recently been removed: https://github.com/googleapis/python-spanner/blob/v1.7.1/setup.py#L32

This warning does not causing docker build timeout. It rather indicates that PyPI is failing to resolve dependency and doing crazy backtrackings

[Abacn]

As a temporary fix Ive had to use poetry instead of pip and it works fine. Would like to know why the dep tree for this using pip forces the installation of google-api-core 2.8.2 which I can see has removed the extra grpcgcp in this pull request of latest version

For released versions, a workaround could be manually install google-cloud-bigquery-storage<2.14 prior installing apache beam:

RUN pip install google-cloud-pubsub==2.13.1
RUN pip install google-cloud-bigquery-storage==2.13.2
RUN pip install apache-beam[gcp]==2.38.0

[NicAlis]

RUN pip install google-cloud-bigquery-storage==2.13.0

This worked for me. Thanks @Abacn.

[NicAlis]

maybe slightly unrelated but if launching the above image as a Dataflow job via Flex Template launcher, the GCE instance setting up the job runs into the same issue (google-api-core 2.8.2 does not provide the extra 'grpcgcp'), even though the SDK image I have built, using the above fix, does not use google-api-core 2.8.2. I have raised an issue with the Dataflow team: https://issuetracker.google.com/issues/238658546.

[Abacn]

Same problem happened again, this time is due to the rollout of google-cloud-pubsub 2.13.2: googleapis/python-pubsub#726

[NicAlis]

I noticed there was something wrong again today but ended up giving up. Any possible fixes for this? a bit annoying that they keep rolling out changes that break stuff, as its holding back some important work for us.

[jkukul]

Another workaround is to use an older version of pip. Older versions (before 21.3) somehow don't force the installation of google-api-core 2.8.2. Or, rather, an older pip seems to be able to faster figure out the working version of dependencies and doesn't time out.

In your Dockerfile change:

RUN pip install --upgrade pip

to:

pip install pip==21.2.4

[NicAlis]

worked, thanks @jkukul!