"No module named pip" during `./install.sh` on macOS, but `pip` / `pip3` is available (IDFGH-4726)

[jschulenklopper]

Environment

Only the relevant items:

• IDF version: v4.4-dev-4-g73db14240
• Build System: cmake 3.19.4
• Operating System: macOS Catalina
• Using an IDE?: Not yet

Problem Description

During step 3 of the ESP-IDF installation on macOS,
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/index.html#id3, the result of ./install.sh is

Installing ESP-IDF tools
Installing tools: xtensa-esp32-elf, xtensa-esp32s2-elf, xtensa-esp32s3-elf, riscv32-esp-elf, esp32ulp-elf, esp32s2ulp-elf, openocd-esp32
Skipping xtensa-esp32-elf@esp-2020r3-8.4.0 (already installed)
Skipping xtensa-esp32s2-elf@esp-2020r3-8.4.0 (already installed)
Skipping xtensa-esp32s3-elf@esp-2020r3-8.4.0 (already installed)
Skipping riscv32-esp-elf@1.24.0.123_64eb9ff-8.4.0 (already installed)
Skipping esp32ulp-elf@2.28.51-esp-20191205 (already installed)
Skipping esp32s2ulp-elf@2.28.51-esp-20191205 (already installed)
Skipping openocd-esp32@v0.10.0-esp32-20200709 (already installed)
Installing Python environment and packages
Creating a new Python environment in /Users/jschulen/.espressif/python_env/idf4.4_py2.7_env
Installing virtualenv
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: No module named pip
Traceback (most recent call last):
  File "/Users/jschulen/Code/iot/esp/esp-idf/tools/idf_tools.py", line 1655, in <module>
    main(sys.argv[1:])
  File "/Users/jschulen/Code/iot/esp/esp-idf/tools/idf_tools.py", line 1651, in main
    action_func(args)
  File "/Users/jschulen/Code/iot/esp/esp-idf/tools/idf_tools.py", line 1357, in action_install_python_env
    stdout=sys.stdout, stderr=sys.stderr)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python', '-m', 'pip', 'install', '--user', 'virtualenv']' returned non-zero exit status 1

Installation of step 1 and 2 went fine. The problem with step 3 seems to be No module named pip but pip / pip3 is available on macOS Catalina:

$ python --version
Python 3.9.1

$ pip --version
pip 21.0.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)

$ which -a python
python: aliased to python3
/usr/bin/python

$ which -a python3
/usr/local/bin/python3
/usr/bin/python3

The problem just before that step is that apparently an environment for Python 2.7 is created or expected. Python 2.7 is still installed on my laptop but not active, Python 3.9 is.

Expected Behavior

An installation that succeeds.

Actual Behavior

See above.

Steps to reproduce

Do the ESP-IDF installation from scratch.

[tanguy-penfen]

Pip dropped python 2 support with version 21. That’s why it can’t find the module. You could try to downgrade or use python 3.
Reading this discussion could help :
#6262
It has some instructions on how to force idf to use python 3.

[jschulenklopper]

But why does it try to use or find a Python 2.7 module? I'm using Python 3 on my laptop: python --version is reporting Python 3.9.1

Referring to a solution in #6262, I did change the shebang in idf_tools.py to force using python3. Then the installation continued without a problem indeed.

So, I'm just not sure why env python is using Python 2.7 on my laptop (but that is outside the scope of this specific issue, I guess).

Oh, I see. The first directory in my $PATH that has a python executable is /usr/bin, and /usr/bin/python is linking to a Python 2.7 version. Apparently I could have prevented this problem if $PATH was set up differently.

Thanks for helping out.

[dobairoland]

I'm sorry for the inconvenience @jschulenklopper. A solution for this is already under way. The installer won't pick "python" if it is of version 2 and will use "python3" if available. The effect will be the same as changing the shebang to "python3".

[jschulenklopper]

It’s fine. The problem is solved - it was also because of my setup (that I was unaware of). The first Python exec on $PATH happened to be Python 2.7 🤦 The bug report can be closed - it’s not really a bug at your side. Thanks for the quick response (from the project)! On 13 Feb 2021, at 11:26, Roland Dobai <notifications@github.com> wrote:  I'm sorry for the inconvenience @jschulenklopper. A solution for this is already under way. The installer won't pick "python" if it is of version 2 and will use "python3" if available. The effect will be the same as changing the shebang to "python3". — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[jenschr]

This problem still occurs. For others that have the same issue - this fix (mentioned above) will solve the problem: #6262 (comment)

[vinifr]

I am getting same error, but the script is using the correct python version, i.e. python 3.8.

Installing Python environment and packages
Python 3.8.2
/home/vini/.espressif/python_env/idf5.0_py3.8_env/bin/python: No module named pip
WARNING: pip is not available in the existing virtual environment, new virtual environment will be created.
WARNING: Removing the existing Python environment in /home/vini/.espressif/python_env/idf5.0_py3.8_env
Creating a new Python environment in /home/vini/.espressif/python_env/idf5.0_py3.8_env
Error: Command '['/home/vini/.espressif/python_env/idf5.0_py3.8_env/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 2.
Traceback (most recent call last):
File "/home/vini/projs/ams_projs/esp32/esp-idf/tools/idf_tools.py", line 2497, in
main(sys.argv[1:])
File "/home/vini/projs/ams_projs/esp32/esp-idf/tools/idf_tools.py", line 2489, in main
action_func(args)
File "/home/vini/projs/ams_projs/esp32/esp-idf/tools/idf_tools.py", line 1975, in action_install_python_env
subprocess.check_call([sys.executable, '-m', 'venv',
File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'venv', '--clear', '/home/vini/.espressif/python_env/idf5.0_py3.8_env']' returned non-zero exit status 1.