west flash: dfu-util.exe not found, unnoticed by require()

[KurtE]

Describe the bug
I am trying build some of the zephyr examples and run them on an Arduino GIGA. In particular some of the
samples within samples/subsystem/input.

The west build completes:

(.venv) D:\zephyrproject\zephyr\samples\subsys\input\input_dump>west build -p -b  arduino_giga_r1//m7 --shield giga_display_shield
-- west build: making build dir D:\zephyrproject\zephyr\samples\subsys\input\input_dump\build pristine
-- west build: generating a build system
Loading Zephyr default modules (Zephyr base).
-- Application: D:/zephyrproject/zephyr/samples/subsys/input/input_dump
-- CMake version: 4.0.1
-- Found Python3: D:/Users/kurte/zephyrproject/.venv/Scripts/python.exe (found suitable version "3.11.9", minimum required is "3.10") found components: Interpreter
-- Cache files will be written to: D:/zephyrproject/zephyr/.cache
-- Zephyr version: 4.1.99 (D:/zephyrproject/zephyr)
-- Found west (found suitable version "1.3.0", minimum required is "0.14.0")
-- Board: arduino_giga_r1, qualifiers: stm32h747xx/m7
-- Shield(s): giga_display_shield
-- ZEPHYR_TOOLCHAIN_VARIANT not set, trying to locate Zephyr SDK
-- Found host-tools: zephyr 0.16.8 (D:/Users/kurte/Downloads/zephyr-sdk-0.16.8)
-- Found toolchain: zephyr 0.16.8 (D:/Users/kurte/Downloads/zephyr-sdk-0.16.8)
-- Found Dtc: C:/ProgramData/chocolatey/bin/dtc.exe (found suitable version "1.5.0", minimum required is "1.4.6")
-- Found BOARD.dts: D:/Users/kurte/zephyrproject/zephyr/boards/arduino/giga_r1/arduino_giga_r1_stm32h747xx_m7.dts                                    -- Found devicetree overlay: D:/zephyrproject/zephyr/boards/shields/arduino_giga_display_shield/giga_display_shield.overlay                          -- Found devicetree overlay: D:/zephyrproject/zephyr/boards/shields/arduino_giga_display_shield/boards/arduino_giga_r1_m7.overlay                    -- Generated zephyr.dts: D:/zephyrproject/zephyr/samples/subsys/input/input_dump/build/zephyr/zephyr.dts                                             -- Generated pickled edt: D:/zephyrproject/zephyr/samples/subsys/input/input_dump/build/zephyr/edt.pickle                                            -- Generated devicetree_generated.h: D:/zephyrproject/zephyr/samples/subsys/input/input_dump/build/zephyr/include/generated/zephyr/devicetree_generated.h                                                                                                                                                 -- Including generated dts.cmake file: D:/zephyrproject/zephyr/samples/subsys/input/input_dump/build/zephyr/dts.cmake                                D:/zephyrproject/zephyr/samples/subsys/input/input_dump/build/zephyr/zephyr.dts:1330.25-1359.5: Warning (spi_bus_bridge): /soc/spi@52005000: incorrect #size-cells for SPI bus                                                                                                                            <stdout>: Warning (spi_bus_reg): Failed prerequisite 'spi_bus_bridge'                                                                                                                                                                                                                                     warning: DISPLAY_INIT_PRIORITY (defined at drivers/display/Kconfig:13) was assigned the value '87'                                                   but got the value ''. Check these unsatisfied dependencies: DISPLAY (=n). See                                                                        http://docs.zephyrproject.org/latest/kconfig.html#CONFIG_DISPLAY_INIT_PRIORITY and/or look up                                                        DISPLAY_INIT_PRIORITY in the menuconfig/guiconfig interface. The Application Development Primer,                                                     Setting Configuration Values, and Kconfig - Tips and Best Practices sections of the manual might be                                                  helpful too.                                                                                                                                                                                                                                                                                                                                                                                                                                                   warning: STM32_LTDC_FB_USE_SHARED_MULTI_HEAP (defined at drivers/display/Kconfig.stm32_ltdc:56) was                                                  assigned the value 'y' but got the value 'n'. Check these unsatisfied dependencies: STM32_LTDC (=n),                                                 DISPLAY (=n). See                                                                                                                                    http://docs.zephyrproject.org/latest/kconfig.html#CONFIG_STM32_LTDC_FB_USE_SHARED_MULTI_HEAP and/or                                                  look up STM32_LTDC_FB_USE_SHARED_MULTI_HEAP in the menuconfig/guiconfig interface. The Application                                                   Development Primer, Setting Configuration Values, and Kconfig - Tips and Best Practices sections of                                                  the manual might be helpful too.                                                                                                                                                                                                                                                                                                                                                                                                                               warning: STM32_LTDC_DISABLE_FMC_BANK1 (defined at drivers/display/Kconfig.stm32_ltdc:59) was                                                         assigned the value 'y' but got the value 'n'. Check these unsatisfied dependencies: STM32_LTDC (=n),                                                 DISPLAY (=n). See                                                                                                                                    http://docs.zephyrproject.org/latest/kconfig.html#CONFIG_STM32_LTDC_DISABLE_FMC_BANK1 and/or look up                                                 STM32_LTDC_DISABLE_FMC_BANK1 in the menuconfig/guiconfig interface. The Application Development                                                      Primer, Setting Configuration Values, and Kconfig - Tips and Best Practices sections of the manual                                                   might be helpful too.                                                                                                                                                                                                                                                                                                                                                                                                                                          warning: The choice symbol STM32_LTDC_RGB565 (defined at drivers/display/Kconfig.stm32_ltdc:38) was                                                  selected (set =y), but no symbol ended up as the choice selection. See                                                                               http://docs.zephyrproject.org/latest/kconfig.html#CONFIG_STM32_LTDC_RGB565 and/or look up                                                            STM32_LTDC_RGB565 in the menuconfig/guiconfig interface. The Application Development Primer, Setting                                                 Configuration Values, and Kconfig - Tips and Best Practices sections of the manual might be helpful                                                  too.                                                                                                                                                                                                                                                                                                      Parsing D:/zephyrproject/zephyr/samples/subsys/input/input_dump/Kconfig                                                                              Loaded configuration 'D:/Users/kurte/zephyrproject/zephyr/boards/arduino/giga_r1/arduino_giga_r1_stm32h747xx_m7_defconfig'                           Merged configuration 'D:/zephyrproject/zephyr/samples/subsys/input/input_dump/prj.conf'                                                              Merged configuration 'D:/zephyrproject/zephyr/boards/shields/arduino_giga_display_shield/boards/arduino_giga_r1_m7.conf'                             Configuration saved to 'D:/zephyrproject/zephyr/samples/subsys/input/input_dump/build/zephyr/.config'                                                Kconfig header saved to 'D:/zephyrproject/zephyr/samples/subsys/input/input_dump/build/zephyr/include/generated/zephyr/autoconf.h'                   -- Found GnuLd: d:/users/kurte/downloads/zephyr-sdk-0.16.8/arm-zephyr-eabi/arm-zephyr-eabi/bin/ld.bfd.exe (found version "2.38")                     -- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- The ASM compiler identification is GNU
-- Found assembler: D:/Users/kurte/Downloads/zephyr-sdk-0.16.8/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc.exe
-- Using ccache: D:/ccache-4.7.4-windows-i686/ccache.exe
-- Configuring done (17.0s)
-- Generating done (0.7s)
-- Build files have been written to: D:/zephyrproject/zephyr/samples/subsys/input/input_dump/build
←[92m-- west build: building application
[1/154] Generating include/generated/zephyr/version.h
-- Zephyr version: 4.1.99 (D:/zephyrproject/zephyr), build: v4.1.0-3063-g16e0adaebc7a
[154/154] Linking C executable zephyr\zephyr.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:       30100 B       768 KB      3.83%
             RAM:        7552 B       512 KB      1.44%
          EXTMEM:          0 GB       256 MB      0.00%
           SRAM1:          0 GB       128 KB      0.00%
           SRAM2:          0 GB       128 KB      0.00%
           SRAM3:          0 GB        32 KB      0.00%
           SRAM4:          0 GB        64 KB      0.00%
          SDRAM1:          0 GB         8 MB      0.00%
        IDT_LIST:          0 GB        32 KB      0.00%
Generating files from D:/zephyrproject/zephyr/samples/subsys/input/input_dump/build/zephyr/zephyr.elf for board: arduino_giga_r1

However, when I try to flash the board (I double click on the reset button to put it into DFU mode...)
The command: west flash
fails:

(.venv) D:\zephyrproject\zephyr\samples\subsys\input\input_dump>west flash
-- west flash: rebuilding
ninja: no work to do.
-- west flash: using runner dfu-util
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\Users\kurte\zephyrproject\.venv\Scripts\west.exe\__main__.py", line 7, in <module>
  File "D:\Users\kurte\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 1176, in main
    app.run(argv or sys.argv[1:])
  File "D:\Users\kurte\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 255, in run
    self.run_command(argv, early_args)
  File "D:\Users\kurte\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 561, in run_command
    self.run_extension(args.command, argv)
  File "D:\Users\kurte\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 716, in run_extension
    self.cmd.run(args, unknown, self.topdir, manifest=self.manifest,
  File "D:\Users\kurte\zephyrproject\.venv\Lib\site-packages\west\commands.py", line 194, in run
    self.do_run(args, unknown)
  File "D:\Users\kurte\zephyrproject\zephyr\scripts\west_commands\flash.py", line 33, in do_run
    do_run_common(self, my_args, runner_args, domain_file=domains_file)
  File "D:\zephyrproject\zephyr\scripts/west_commands\run_common.py", line 349, in do_run_common
    prev_runner = do_run_common_image(command, user_args, user_runner_args, used_cmds,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\zephyrproject\zephyr\scripts/west_commands\run_common.py", line 523, in do_run_common_image
    runner = runner_cls.create(runner_config, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\zephyrproject\zephyr\scripts/west_commands\runners\core.py", line 680, in create
    ret = cls.do_create(cfg, args)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\zephyrproject\zephyr\scripts/west_commands\runners\dfu.py", line 94, in do_create
    ret.ensure_device()
  File "D:\zephyrproject\zephyr\scripts/west_commands\runners\dfu.py", line 98, in ensure_device
    if not self.find_device():
           ^^^^^^^^^^^^^^^^^^
  File "D:\zephyrproject\zephyr\scripts/west_commands\runners\dfu.py", line 106, in find_device
    output = self.check_output(cmd)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "D:\zephyrproject\zephyr\scripts/west_commands\runners\core.py", line 887, in check_output
    return subprocess.check_output(cmd, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Program Files\Python311\Lib\subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Program Files\Python311\Lib\subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Program Files\Python311\Lib\subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "D:\Program Files\Python311\Lib\subprocess.py", line 1538, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified

(.venv) D:\zephyrproject\zephyr\samples\subsys\input\input_dump>

If I run these same commands on my Ubuntu machine, the west flash succeeds.

Also note: With my Windows 11 machine. I have successfully built and programmed some
Teensy boards (PJRC: teensy40 teensy41 and teensymm with my PR code in place)

I am running Windows 11 Home 24H2 on DELL with Intel 12th generation I7...

To Reproduce

1. Install zephyr on on Windows, have venv active.
2. cd to the test directory: in my case:
   d:
   cd zephyrproject\zephyr\samples\subsys\input\input_dump
3. west build -p -b arduino_giga_r1//m7 --shield giga_display_shield
4. west flash
5. See error-->

Expected behavior
I expect it to write the program to the GIGA.
Impact
Stops me from using my Windows install, to try out stuff directly on Zephyr

I can still build the ArduinoCore-zephyr setup (Arduino with zephyr) on my windows machine,
by building the zephyr loader on WSL, copy the stuff back into windows setup and use the Arduino IDE
setup to flash... (Might be able to get that command to work for these programs by copying the built
hex file into the place that the Arduino command for burn bootloader is expecting the file... May try)

Logs and console output

Environment (please complete the following information):

• OS: Windows
• Toolchain installed using instructions: https://docs.zephyrproject.org/latest/develop/getting_started/index.html
• zephyr is pretty close to up to data: currently using my PR branch: boards: Support Teensy Micromod #88502
• sdk: 16.8 I think

Additional context

[KurtE]

Quick note: I was able to use the dfu-util from Arduino 2.x install:

(.venv) D:\zephyrproject\zephyr_test_sketches\print_touch_events>"C:\Users\kurte\AppData\Local\Arduino15\packages\arduino\tools\dfu-util\0.10.0-arduino1/dfu-util" --device 0x2341:0x0366 -D "build\zephyr\zephyr.bin" -a0 --dfuse-address=0x08040000:leave
dfu-util 0.10-dev

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2021 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

Warning: Invalid DFU suffix signature
A valid DFU suffix will be required in a future dfu-util release
Opening DFU capable USB device...
Device ID 2341:0366
Device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Interface #0 ...
Determining device status...
DFU state(2) = dfuIDLE, status(0) = No error condition is present
DFU mode device DFU version 011a
Device returned transfer size 4096
DfuSe interface name: "Internal Flash  2MB   "
Downloading element to address = 0x08040000, size = 38280
Erase           [=========================] 100%        38280 bytes
Erase    done.
Download        [=========================] 100%        38280 bytes
Download done.
File downloaded successfully
Transitioning to dfuMANIFEST state

[josuah]

I am not very familiar with Windows deployments, however:

C:\Users\kurte\AppData\Local\Arduino15\packages\arduino\tools\dfu-util\0.10.0-arduino1/

Is this directory in the current binary search path?

If adding it fixes the bug, this might be a documentation bug, or unhelpful error message. Suggesting the user to check the presence for the dfu-util program in the current PATH could address that maybe.

[KurtE]

Is this directory in the current binary search path?

It turns out is not in the search path on my Windows machine. Arduino installs it's over version(s) depending on
what each board/platform says they require.

It now works, that I copied a version of it to my user bin file (c:\users\kurte\bin) which is on my search path

What confused me, is that it just worked fine on my Ubuntu machine, but it turns out I already had a copy
of it in my /usr/bin/ directory. I probably copied or downloaded it there a while ago, to probably flash some other
board.

Not sure if I just close this out, or somewhere there should be documentation about this?
If not feel free to close it.

[josuah]

Very glad to hear that!
All Arduino users trying Zephyr might face this same error.

It looks like a bug as the error messages should have been slightly different:

zephyr/scripts/west_commands/runners/core.py

Lines 795 to 812 in d26ed34

	def require(program: str, path: str | None = None) -> str:
	'''Require that a program is installed before proceeding.
	:param program: name of the program that is required,
	or path to a program binary.
	:param path: PATH where to search for the program binary.
	By default check on the system PATH.
	If ``program`` is an absolute path to an existing program
	binary, this call succeeds. Otherwise, try to find the program
	by name on the system PATH or in the given PATH, if provided.
	If the program can be found, its path is returned.
	Otherwise, raises MissingProgram.'''
	ret = shutil.which(program, path=path)
	if ret is None:
	raise MissingProgram(program)
	return ret

P.S.: It seems possible to specify a --dfu-util=...path\to\dfu-util.exe:

zephyr/scripts/west_commands/runners/dfu.py

Lines 76 to 77 in d26ed34

	parser.add_argument('--dfu-util', default='dfu-util',
	help='dfu-util executable; defaults to "dfu-util"')

[josuah]

Let me know if the new name for the issue is not fitting!

[KurtE]

Sounds good... Thanks

[pdgendt]

Sorry, I don't have experience with this. Maybe @carlescufi can help?