Issues with `PYTHONPATH` resolution in recent python/rules_python versions

[phlax]

🐞 bug report

Affected Rule

The issue is caused by the rule:

afaict ~any python rules but ive tested or hit the issue with py_binary, py_library

Is this a regression?

Yes, the previous version in which this bug was not present was: ....

We had trouble upgrading to 0.19.x versions iirc due to this issue

We have been able to upgrade to 0.20.0 but when upgrading the python version to 3.11 the issue reappears

Description

A clear and concise description of the problem...

Expected paths are not being added to sys.path

🔬 Minimal Reproduction

py_library(
    name = "mylib",
    srcs = ["mylib.py"],
)

py_binary(
    name = "mybin",
    srcs = ["mybin.py"],
    deps = [":mylib"],
)

def myfun():
    print("BOOM!")

from mylib import myfun


def main():
    myfun()


if __name__ == "__main__":
    main()

🔥 Exception or Error

$ bazel run //tools/base:mybin
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 19b6db42-4a15-4858-be40-7cf804c798de
INFO: Analyzed target //tools/base:mybin (126 packages loaded, 3458 targets configured).
INFO: Found 1 target...
Target //tools/base:mybin up-to-date:
  bazel-bin/tools/base/mybin
INFO: Elapsed time: 4.511s, Critical Path: 0.28s
INFO: 4 processes: 4 internal.
INFO: Build completed successfully, 4 total actions
INFO: Running command line: bazel-bin/tools/base/mybin
Traceback (most recent call last):
  File "/home/worker/.cache/bazel/_bazel_worker/f704bab1b165ed1368cb88f9f49e7532/execroot/envoy/bazel-out/k8-fastbuild/bin/tools/base/mybin.runfiles/envoy/tools/base/mybin.py", line 2, in <module>
    from mylib import myfun
ModuleNotFoundError: No module named 'mylib'

🌍 Your Environment

Operating System:

  
Linux
  

Output of bazel version:

  
$ bazel version
Bazelisk version: v1.11.0
Build label: 6.1.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Mar 6 17:09:47 2023 (1678122587)
Build timestamp: 1678122587
Build timestamp as int: 1678122587

  

Rules_python version:

  
0.20.0
  

Anything else relevant?

Python version: 3.11

[phlax]

cc @htuch

[phlax]

adding a shim - ie:

import os
import sys

sys.path += [os.path.dirname(__file__)]

works, and is what i have done to workaround, either in our py targets, or for py files in dep rules

[kevinaud]

@phlax Can you elaborate on how you hooked that shim in? I think I'm currently running into this same issue. I'm on rules_python version 0.21.0 and python 3.11.1.

Things are working on my dev machine but during my automated build/tests in github actions I'm getting this (I have a python tool for running these tests which is why the logger prefix is present):

INFO: Repository third_party_docopt instantiated at:
2023-06-09 11:37:53,037 - tools.presubmit.cli - INFO - /home/runner/work/app/app/WORKSPACE:100:13: in <toplevel>
2023-06-09 11:37:53,037 - tools.presubmit.cli - INFO - /home/runner/.cache/bazel/_bazel_runner/9bb8a7cb6[522](https://github.com/company/app/actions/runs/5221416166/jobs/9425648690#step:12:523)6a357f750dd2451a3938/external/third_party/requirements.bzl:47:20: in install_deps
2023-06-09 11:37:53,037 - tools.presubmit.cli - INFO - Repository rule whl_library defined at:
2023-06-09 11:37:53,037 - tools.presubmit.cli - INFO - /home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/rules_python/python/pip_install/pip_repository.bzl:712:30: in <toplevel>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - ERROR: An error occurred during the fetch of repository 'third_party_docopt':
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Traceback (most recent call last):
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/rules_python/python/pip_install/pip_repository.bzl", line 688, column 13, in _whl_library_impl
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - fail("whl_library %s failed: %s (%s)" % (rctx.attr.name, result.stdout, result.stderr))
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Error in fail: whl_library third_party_docopt failed: Collecting docopt==0.6.2
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Downloading docopt-0.6.2.tar.gz (25 kB)
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Preparing metadata (setup.py): started
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Preparing metadata (setup.py): finished with status 'error'
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - (  error: subprocess-exited-with-error
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - 
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - × python setup.py egg_info did not run successfully.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - │ exit code: 1
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - ╰─> [6 lines of output]
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Traceback (most recent call last):
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "<string>", line 2, in <module>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "<pip-setuptools-caller>", line 34, in <module>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/tmp/pip-wheel-yuurxaqt/docopt_0109a8f2a8f74fe78074f6032e6f0359/setup.py", line 3, in <module>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - from docopt import __version__
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - ModuleNotFoundError: No module named 'docopt'
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - [end of output]
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - 
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - note: This error originates from a subprocess, and is likely not a problem with pip.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - error: metadata-generation-failed
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - 
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - × Encountered error while generating package metadata.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - ╰─> See above for output.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - 
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - note: This is an issue with the package mentioned above, not pip.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - hint: See above for details.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Traceback (most recent call last):
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "<frozen runpy>", line 198, in _run_module_as_main
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "<frozen runpy>", line 88, in _run_code
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 450, in <module>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - main()
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 427, in main
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - subprocess.run(pip_args, check=True, env=env)
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/python3_11_x86_64-unknown-linux-gnu/lib/python3.11/subprocess.py", line 571, in run
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - raise CalledProcessError(retcode, process.args,
2023-06-09 11:37:53,047 - tools.presubmit.cli - INFO - subprocess.CalledProcessError: Command '['/home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/python3_11_x86_64-unknown-linux-gnu/bin/python3', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '-r', '/tmp/tmpck70_4v8']' returned non-zero exit status 1.
2023-06-09 11:37:53,047 - tools.presubmit.cli - INFO - )
2023-06-09 11:37:53,047 - tools.presubmit.cli - INFO - ERROR: /home/runner/work/app/app/WORKSPACE:100:13: fetching whl_library rule //external:third_party_docopt: Traceback (most recent call last):

Not these lines in particular:

2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/tmp/pip-wheel-yuurxaqt/docopt_0109a8f2a8f74fe78074f6032e6f0359/setup.py", line 3, in <module>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - from docopt import __version__
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - ModuleNotFoundError: No module named 'docopt'

Can't figure out how to resolve this for the life of me.

[phlax]

Can you elaborate on how you hooked that shim in?

very hackily - see this PR for details https://github.com/envoyproxy/envoy/pull/25359/files

seems to have survived a rules_python upgrade, i didnt check to see if the hacks could be removed yet

[alexeagle]

I expect this is resolved by https://github.com/aspect-build/rules_py because it doesn't mess with the PYTHONPATH, it just lays out a python-idiomatic virtualenv with a site-packages folder.

[phlax]

interesting we just landed a workaround to inject namespacing init pys to our repo folders to avoid pypi namespace clashing - seems like we could avoid that workaround - ill look further when i get chance

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[phlax]

this is still an issue

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[alexeagle]

Note, I deleted the stale bot :)