CMake does not use OS Version of WxWidgets

[msglm]

The CMake file always tries to download wxWidgets from the git source rather than using the OS version. This makes packaging this software for GNU Guix SD much harder as Guix (the package manager) is source based and cannot download any new code for the source code when compiling so the code always remains reproducible.

Adding a check to see if wxwidgets is available locally in the cmake should be done to prevent this problem. This may help with packaging on other distributions as well as they can link their compiled binaries against their already packaged version of WxWidgets.

[fszontagh]

If you can, please testout with the current main branch.
I added today two new cmake parameters:

• WXWIDGETS_EXTRA_PATH
• WXWIDGETS_VERSION

This two parameters are independent themselves.
The WXWIDGETS_EXTRA_PATH is used to specify a path, where the wxwidgets is available. (If the wxwidgets is not in the system paths, eg. cmake --install . --prefix /opt/wxWidgets-3.2.x then -DWXWIDGETS_EXTRA_PATH=/opt/wxWidgets-3.2.x)

The second parameter is for to force a wxWidgets version. The default version is 3.2.6, as i saw, the GNU Guix OS have the wxWidgets 3.2.5 available.

I prefer the latest wxWidgets version, so this is the reason why i forced to compile it on Ubuntu. On windows, this version in available with conan.

I am curious, so if you have .scm file for it, i would like to try it.

[msglm]

[ 11%] Creating directories for 'wxwidgets-populate'
[ 22%] Performing download step (git clone) for 'wxwidgets-populate'
Cloning into 'wxwidgets-src'...
fatal: unable to access 'https://github.com/wxWidgets/wxWidgets.git/': Could not resolve host: github.com
Cloning into 'wxwidgets-src'...
fatal: unable to access 'https://github.com/wxWidgets/wxWidgets.git/': Could not resolve host: github.com
Cloning into 'wxwidgets-src'...
fatal: unable to access 'https://github.com/wxWidgets/wxWidgets.git/': Could not resolve host: github.com
-- Had to git clone more than once: 3 times.
CMake Error at wxwidgets-subbuild/wxwidgets-populate-prefix/tmp/wxwidgets-populate-gitclone.cmake:39 (message):
  Failed to clone repository: 'https://github.com/wxWidgets/wxWidgets.git'


make[2]: *** [CMakeFiles/wxwidgets-populate.dir/build.make:102: wxwidgets-populate-prefix/src/wxwidgets-populate-stamp/wxwidgets-populate-download] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/wxwidgets-populate.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

CMake Error at /gnu/store/pb978182ffdf5p8j1w97bziph1fy31yg-cmake-minimal-3.24.2/share/cmake-3.24/Modules/FetchContent.cmake:1604 (message):
  Build step for wxwidgets failed: 2
Call Stack (most recent call first):
  /gnu/store/pb978182ffdf5p8j1w97bziph1fy31yg-cmake-minimal-3.24.2/share/cmake-3.24/Modules/FetchContent.cmake:1744:EVAL:2 (__FetchContent_directPopulate)
  /gnu/store/pb978182ffdf5p8j1w97bziph1fy31yg-cmake-minimal-3.24.2/share/cmake-3.24/Modules/FetchContent.cmake:1744 (cmake_language)
  /gnu/store/pb978182ffdf5p8j1w97bziph1fy31yg-cmake-minimal-3.24.2/share/cmake-3.24/Modules/FetchContent.cmake:1958 (FetchContent_Populate)
  cmake/wxWidgets.cmake:17 (FetchContent_MakeAvailable)
  CMakeLists.txt:49 (include)

here's an excerpt from the .scm file:

(define-public exiv2-staticlibs
               (package
                 (inherit exiv2)
    (arguments
     `(#:test-target "tests"
       #:configure-flags (list "-DEXIV2_BUILD_UNIT_TESTS=ON"
                               ;; darktable needs BMFF to support
                               ;; CR3 files.
                               "-DEXIV2_ENABLE_BMFF=ON")
       #:phases
       (modify-phases %standard-phases
         (add-after 'unpack 'patch-gcc-reference
           (lambda _
             ;; _GLIBCXX_ASSERTIONS brings reference to GCC.
             (substitute* "cmake/compilerFlags.cmake"
               (("add_compile_options[(]-Wp,-D_GLIBCXX_ASSERTIONS[)]")
                ""))))

         ,@(if (or (target-ppc64le?)
                   (target-aarch64?)
                   (target-riscv64?))
               '((add-after 'unpack 'adjust-tests
                   (lambda _
                     ;; Adjust test on ppc64 and aarch64, where no exception
                     ;; is raised and thus the return value is different.  See
                     ;; <https://github.com/Exiv2/exiv2/issues/365> and
                     ;; <https://github.com/Exiv2/exiv2/issues/933>.
                     (substitute* "tests/bugfixes/github/test_CVE_2018_12265.py"
                       (("\\$uncaught_exception \\$addition_overflow_message\n") "")
                       (("retval = \\[1\\]") "retval = [0]")))))
               '()))))
                 ))

(define-public stable-diffusion-cpp-gui-wx
               (package
                        (name "stable-diffusion-cpp-gui-wx")
                        (version "6f604f836961002fbce72a205c1e6cc6d6a531e9")
                        (source
                          (origin
                            (method git-fetch)
                                    (uri (git-reference
                                           (url "https://github.com/fszontagh/sd.cpp.gui.wx")
                                           (recursive? #t)
                                           (commit version)))
                                    (file-name (git-file-name name version))
                                    (sha256
                                      (base32
                                        "035zqhjnwdb07ag4chj9zijl70wk81kinqwlnvs9ixb2fzl1z374"))))
                 (build-system cmake-build-system)
                 (arguments
                   '(
                     #:tests? #f
                     #:configure-flags
                     (list 
                       "-DBUILD_SHARED_LIBS=ON"
                       (string-append "-DWXWIDGETS_EXTRA_PATH=" (assoc-ref %build-inputs "wxwidgets"))
                       (string-append "-DWXWIDGETS_VERSION=" (assoc-ref %build-inputs "wxwidgets"))

                           )
                     #:substitutable? #f
                     )
                   )
               (native-inputs (list pkg-config))
               (inputs (list openblas openssl curl exiv2-staticlibs wxwidgets gtk+ pangomm git))
               (synopsis "cross-platform GUI for Stable Diffusion C++")
               (description "A cross-platform GUI for Stable Diffusion C++, built using wxWidgets.")
               (home-page "https://stable-diffusion.fsociety.hu")
               (license license:expat)
               ))

I prefer the latest wxWidgets version

If it necessitates it, I can just bump the version or make an alternative version just for this. Still need to have it compile in the first place to even test if it works with 3.2.5.

[fszontagh]

I tested the wxWidgets 3.2.5 and it's fine on Ubuntu24.
In theory all the 3.x.x is compatible

The WXWIDGETS_VERSION is a version string, e.g.: 3.2.5, not an entire path
The WXWIDGETS_EXTRA_PATH is required only when cmake can not find the wxwidgets in the system paths, this path will used as additional path.
The BUILD_SHARED_LIBS=ON is not used, not required here, handled internally.

For a running copy, there is must be at least one backend parameter to build, for example:

-DSD_AVX=ON

Otherwise no shared library compiled from the sd.cpp, but at least one is required.

So, this is separated in the cmake into multiple targets, after the configure is done.

For example:

cmake .. -DSD_AVX=ON

Then a target created with name stable_diffusion_cpp_avx

This is because the packaging systems. In ubuntu, the libs are packaged into separated .deb files, to keep the sizes down.

So i tried with this:

(use-modules (guix packages)
             (guix download)
             (guix build-system cmake)
             (guix licenses)
             (guix git-download)
             (guix utils)
             (gnu packages version-control)
             (gnu packages pkg-config)
             (gnu packages image)
             (gnu packages tls)
             (gnu packages maths)
             (gnu packages curl)
             (gnu packages image)
             (gnu packages wxwidgets)
             (gnu packages gtk))

(define-public exiv2-staticlibs
               (package
                 (inherit exiv2)
    (arguments
     `(#:test-target "tests"
       #:configure-flags (list "-DEXIV2_BUILD_UNIT_TESTS=ON"
                               ;; darktable needs BMFF to support
                               ;; CR3 files.
                               "-DEXIV2_ENABLE_BMFF=ON")
       #:phases
       (modify-phases %standard-phases
         (add-after 'unpack 'patch-gcc-reference
           (lambda _
             ;; _GLIBCXX_ASSERTIONS brings reference to GCC.
             (substitute* "cmake/compilerFlags.cmake"
               (("add_compile_options[(]-Wp,-D_GLIBCXX_ASSERTIONS[)]")
                ""))))

         ,@(if (or (target-ppc64le?)
                   (target-aarch64?)
                   (target-riscv64?))
               '((add-after 'unpack 'adjust-tests
                   (lambda _
                     ;; Adjust test on ppc64 and aarch64, where no exception
                     ;; is raised and thus the return value is different.  See
                     ;; <https://github.com/Exiv2/exiv2/issues/365> and
                     ;; <https://github.com/Exiv2/exiv2/issues/933>.
                     (substitute* "tests/bugfixes/github/test_CVE_2018_12265.py"
                       (("\\$uncaught_exception \\$addition_overflow_message\n") "")
                       (("retval = \\[1\\]") "retval = [0]")))))
               '()))))
                 ))
  (package
    (name "stable-diffusion-cpp-gui-wx")
    (version "6f604f836961002fbce72a205c1e6cc6d6a531e9")
    (source
      (origin
        (method git-fetch)
        (uri (git-reference
               (url "https://github.com/fszontagh/sd.cpp.gui.wx")
               (commit version)))
        (file-name (git-file-name name version))
        (sha256
          (base32
            "035zqhjnwdb07ag4chj9zijl70wk81kinqwlnvs9ixb2fzl1z374"))))
    (build-system cmake-build-system)
    (arguments
      `(#:tests? #f
        #:configure-flags
        (list
          "-DSD_AVX=ON"
          (string-append "-DWXWIDGETS_EXTRA_PATH=" (assoc-ref %build-inputs "wxwidgets"))
          (string-append "-DWXWIDGETS_VERSION=3.2.5"))))
    (native-inputs (list pkg-config))
    (inputs (list openblas openssl curl exiv2-staticlibs wxwidgets gtk+ pangomm git))
    (synopsis "Cross-platform GUI for Stable Diffusion C++")
    (description "A cross-platform GUI for Stable Diffusion C++, built using wxWidgets.")
    (home-page "https://stable-diffusion.fsociety.hu")
    (license gpl3+))

With an environment:

guix environment --network --ad-hoc git curl

But got the same error at FetchContent.

So i checked out the directory, which is coming from here:

(string-append "-DWXWIDGETS_EXTRA_PATH=" (assoc-ref %build-inputs "wxwidgets"))

Without the environment change, i got an old wxWidgets version, with the environment, i got the 3.2.5. But there is no cmake config files in the compiled cmake directory, so the sdgui's cmake started to download it from git.

Because i don't know this OS and this package manager, i can't go further.

But in packaging, it's required to separate this project into multiple packages. You can investigate the github workflow to see how it's works.

[fszontagh]

Ok, i can't go throught the wxWidgets, so it's maybe need to add as like the "define-public exiv2-staticlibs", and compile it too, but the wxWidgets liblaries are linked as statically too into the binaries. So i think this is the way.

If the wxWidgets is fine, then need to resolve the issue with the sd.cpp libs, because it's uses FetchContent too. But sd.cpp is linked dynamically as i mentioned earlier.

As i saw in other scm files, everywhere is 'patched' out the FetchContent thing from the cmake files.

Maybe it's possible to use these libs as git submodule, but this is a larger refactoring thing in this project.

This is the latest scm what i tried:

(use-modules (guix packages)
             (guix download)
             (guix build-system cmake)
             (guix licenses)
             (guix git-download)
             (guix utils)
             (gnu packages version-control)
             (gnu packages pkg-config)
             (gnu packages image)
             (gnu packages tls)
             (gnu packages maths)
             (gnu packages curl)
             (gnu packages image)
             (gnu packages wxwidgets)
             (gnu packages gtk))

(define-public exiv2-staticlibs
               (package
                 (inherit exiv2)
    (arguments
     `(#:test-target "tests"
       #:configure-flags (list "-DEXIV2_BUILD_UNIT_TESTS=ON"
                               ;; darktable needs BMFF to support
                               ;; CR3 files.
                               "-DEXIV2_ENABLE_BMFF=ON")
       #:phases
       (modify-phases %standard-phases
         (add-after 'unpack 'patch-gcc-reference
           (lambda _
             ;; _GLIBCXX_ASSERTIONS brings reference to GCC.
             (substitute* "cmake/compilerFlags.cmake"
               (("add_compile_options[(]-Wp,-D_GLIBCXX_ASSERTIONS[)]")
                ""))))

         ,@(if (or (target-ppc64le?)
                   (target-aarch64?)
                   (target-riscv64?))
               '((add-after 'unpack 'adjust-tests
                   (lambda _
                     ;; Adjust test on ppc64 and aarch64, where no exception
                     ;; is raised and thus the return value is different.  See
                     ;; <https://github.com/Exiv2/exiv2/issues/365> and
                     ;; <https://github.com/Exiv2/exiv2/issues/933>.
                     (substitute* "tests/bugfixes/github/test_CVE_2018_12265.py"
                       (("\\$uncaught_exception \\$addition_overflow_message\n") "")
                       (("retval = \\[1\\]") "retval = [0]")))))
               '()))))
                 ))
  (package
    (name "stable-diffusion-cpp-gui-wx")
    (version "main")
    (source
      (origin
        (method git-fetch)
        (uri (git-reference
               (url "https://github.com/fszontagh/sd.cpp.gui.wx")
               (commit version)))
        (file-name (git-file-name name version))
        (sha256
          (base32
            "0npsxglr8a9j2ysi4635pmvawk6vv9gw10h5sc0m65fjn8508h7w"))))
    (build-system cmake-build-system)
                 (arguments
                   '(
                     #:tests? #f
                     #:configure-flags
                     (list
                       "-DSD_AVX=ON"
                       "-DWXWIDGETS_PKGCONF=OFF"
                           )
                     #:substitutable? #f
                     )
                   )
    (native-inputs (list pkg-config))
    (inputs (list openblas openssl curl exiv2-staticlibs wxwidgets gtk+ pangomm git))
    (synopsis "Cross-platform GUI for Stable Diffusion C++")
    (description "A cross-platform GUI for Stable Diffusion C++, built using wxWidgets.")
    (home-page "https://stable-diffusion.fsociety.hu")
    (license gpl3+))

(FYI: the license is wrong, this project is using the MIT license)

I think it is a good idea to separate the sd.cpp libs build from the GUI's build like i do in the debian packages.
The cuda, rocm and the vulkan variants have different dependencies, which is easier to handle in separated packages. And for the ROCM variant needs the ninja and clang tools.
Another thing is the cpu_features which using FetchContent too. But i found it in the GUIX repo.

[msglm]

there is no cmake config files in the compiled cmake directory, so the sdgui's cmake started to download it from git.

I checked the wxwidgets package on my end and can confirm as well that there is just are no .cmake files in the compiled directory. Usually, something would be under lib/cmake.

This is a problem with the WxWidgets definition in Guix or the WxWidgets project itself and your code has no fault in this. I'll see about getting this fixed somehow.

I think it is a good idea to separate the sd.cpp libs build from the GUI's build like i do in the debian packages

I do have a different package definition for sd.cpp that I can just declare as a dependency. I'm trying to get a GUI for this set up (along with a free software GPU accelerator of some kind) first before submitting it upstream.

Here's the package definition:

(define-public stable-diffusion-cpp
               (package
                 (name "stable-diffusion-cpp")
                 (version "9578fdcc4632dc3de5565f28e2fb16b7c18f8d48")
                 (source
                   (origin
                     (method git-fetch)
                     (uri (git-reference
                            (url "https://github.com/leejet/stable-diffusion.cpp.git")
                            (recursive? #t)
                            (commit version)))
                     (file-name (git-file-name name version))
                     (sha256
                       (base32
                         "0dhh4m710kiwrs4yc64saab5rkjg5psxr15cm6fsl64pv4ssk7v5"))))
                 (build-system cmake-build-system)
                 (inputs (list openblas))
                 (arguments
                   '(
                     #:tests? #f
                     #:configure-flags (list "-DGGML_OPENBLAS=ON -DSD_FLASH_ATTN=ON --config Release")
                     )
                   )
                 (home-page "https://github.com/leejet/stable-diffusion.cpp")
                 (synopsis "Inference of Stable Diffusion in pure C/C++")
                 (description "Inference of Stable Diffusion in pure C/C++")
                 (license license:expat)))

it's required to separate this project into multiple packages

I'm assuming for Rocm, CPU, and CUDA support, correct? Guix has an inheriting system so it is rather trivial to make support for build variations.

I would like to get Vulkan support running since it's nearly universal and can run on free drivers, but it requires some very jank pre-compiling of shaders and source-code modifications per-compile to function properly. It's not very robust so I only run it on my personal system in my guix home definition.

Maybe it's possible to use these libs as git submodule, but this is a larger refactoring thing in this project

This would be the superior way of doing this if the dependencies are to be pulled in. That way the build is always reproducible so long as you grab the code from the source.

Furthermore, I believe this will make the in-code libraries a fallback if there isn't a system package that has them. I am not the most familiar with CMake, but I think it prefers system libraries to in-code local libraries.

the license is wrong, this project is using the MIT license

I have just changed it, it was just some scheme code I copied to get everything working.

[fszontagh]

This is a problem with the WxWidgets definition in Guix

If i compile wxWidgets (with cmake) on Ubuntu, the libs/cmake is always placed into to the good place.
(cmake --install . --prefix /opt/...).

I do have a different package definition for sd.cpp that I can just declare as a dependency. I'm trying to get a GUI for this set up (along with a free software GPU accelerator of some kind) first before submitting it upstream.

Please take care to the sd.cpp and the GUI compatibility. The current main branch of the GUI is using the 'master-b5f4932' from the sd.cpp, which is older than the current. So, the current sd.cpp (master-9578fdc) is incompatible with the GUI.

This file contains the current used sd.cpp "version". Either older or newer version is not compatible.

I'm assuming for Rocm, CPU, and CUDA support, correct?

Yes

I would like to get Vulkan support running since it's nearly universal and can run on free drivers, but it requires some very jank pre-compiling of shaders and source-code modifications per-compile to function properly. It's not very robust so I only run it on my personal system in my guix home definition.

In the current used sd.cpp "version" (b5f4932), the vulkan build is failing. In the latest sd.cpp version (9578fdc) the ROCM build is failing.

There is a lot of PR, witch is waiting to merge at the sd.cpp project. These PRs will fix (maybe) all variations. (and adding a lot of new wonderful things, e.g. the in and out painting :) ). So i currently waiting for the new sd.cpp release to update the GUI to it.

This would be the superior way of doing this if the dependencies are to be pulled in. That way the build is always reproducible so long as you grab the code from the source.

CMake's fetch content and the git submodule is nearly the same thing. At least the end result is same. But i think the submodules harder to manage. And the submodules need to be cloned even the package exists in the system.

Furthermore, I believe this will make the in-code libraries a fallback if there isn't a system package that has them. I am not the most familiar with CMake, but I think it prefers system libraries to in-code local libraries.

Good point. But as we mentioned earlier, no cmake config files in the GNU Guix's wxWidgets package.

[msglm]

the libs/cmake is always placed into to the good place

Interesting, I will have to research more into this and build WxWidgets from source without using the package manager to see what is happening.

I think that something about how the build system is configured in Guix is breaking the production of lib/cmake. Likely some flag not being set by default or some folder not being copied that should be.

Please take care to the sd.cpp and the GUI compatibility

Acknowledged. I figure by the time I get this working there will be a new update that will let this project use the latest sd.cpp code. If not, I will just create a variant for this project that sticks to the version of sd.cpp this gui is developed for.

But i think the submodules harder to manage. And the submodules need to be cloned even the package exists in the system

If you find yourself more comfortable with the your current system, don't change it. I will try to find a work around for what I am doing. Guix is very capable of handling differences in maintainer workflows