Support Modular Framework modulemap

[frogcjn]

What is the problem or limitation you are having?

As discussed in #249

we should update the location of module.modulemap and its content to fit the preference of Darwin-styled framework.
it will support Xcode automatically mapping to Xcode's Framework Search Paths (-F) instead of changing Build Settings of Header Search Paths.

Describe the solution you'd like

1. We should update the module.modulemap into the right location into Python.framework/Modules folder
   and update its content to:

framework module Python {
    umbrella header "Python.h"
    export *
    link "Python"
}

2. edit the content in the file Headers/cpython/pyatomic.h:

   • replace cpython/pyatomic_gcc.h with pyatomic_gcc.h
   • replace cpython/pyatomic_std.h with pyatomic_std.h
   • replace cpython/pyatomic_msc.h with pyatomic_msc.h

3. Delete the setting of Target - Build settings - Search path - Header Search Paths in Xcode

Describe alternatives you've considered

Keep current design, but notice developer to adjust Target - Build settings - Search path - Header Search Paths to $(BUILT_PRODUCTS_DIR)/Python.framework/Headers in Xcode.

Additional context

the difference between framework module a non-framework module is:
https://chatgpt.com/share/67b02a23-00d8-8004-abdc-f21bf070b7c5

the difference between framework modulemap and non-framework modulemap module is:
https://chatgpt.com/share/67b029dd-5ae4-8004-a4ab-7a0c37db8f11

The right location of framework modulemap is also documented on clang website:

1. make sure the framework modulemap file in the Python.framework/Modules folder has framework keyword and link "python" line as:

framework module Python {
    umbrella header "Python.h"
    export *
    link "Python"
}

2. Delete the setting of Target - Build settings - Search path - Header Search Paths in Xcode

3. Click Product -> Clear Build Folder... and build it again,
   the 100 warnings about headers will disappear on macOS.

I have tried several times upon my own instructions on macOS
https://www.notion.so/frogcjn/Swift-with-Python-Step-by-Step-1984959764ca80999f25cf9897df83bc

[freakboy3742]

As noted in previous comments, part (2) of this proposal isn't viable.

Providing ChatGPT responses as "explanations" of an API isn't helpful, as we have no way to validate whether the responses are accurate. ChatGPT produces content that looks like an answer to any question, but will not ever say "I don't know", and will fabricate information to satisfy its requirement for "answer like" content.

I don't see the headers disappear after cleaning and rebuilding the project. Based on what I am reading about the "Umbrella header" warning, I don't see how you're not seeing the warnings unless you're explicitly disabling the -Wincomplete-umbrella warnings at the compiler level. I am testing on iOS, rather than macOS, but that doesn't seem to be an obvious cause.

However, if I add lines like:

     exclude header "marshal.h"
     exclude header "py_curses.h"
     exclude header "errcode.h"

to the modulemap, the warnings are silenced. It looks like it could also be fixed by moving those headers into a PrivateHeaders folder, as with the pyatomics renaming, that would be much more invasive.

[frogcjn]

https://samsymons.com/blog/understanding-objective-c-modules/
framework module is used for framework
module is used for static library and sub module in framework

Could you tell me how to turn -Wincomplete-umbrella on? I'm using Xcode to compile the macOS/iOS target.

Could you tell me your Xcode version? and Project's target macOS version?
My project is not setting this, but get no warnings.

[freakboy3742]

I haven't done anything to turn -Wincomplete-umbrella on - it's on by default (on iOS at least).

I'm running on Xcode 16.2, on an iOS 18.2 project. I haven't tested macOS yet; it's possible this might be a quirk of iOS compilation.

[frogcjn]

@freakboy3742
I have created an iOS project, also follow up this modulemap settings, and Python official website settings, and delete the setting of Target - Build settings - Search path - Header Search Paths in Xcode, still no warnings, both on simulator and device.

Are you working on Apple Silicon Mac or Intel based Mac?

Maybe you could give me your Xcode project I'll investigate it.
Also you could try my whole project to see whether my project will have warning on your machine: https://github.com/frogcjn/SwiftwithPython

[freakboy3742]

I'm on an M1 MacBook Pro, running macOS Sequoia 15.2.

I've just reproduced the problem on a macOS project; you can download the project here. This project uses the published 3.13 support package, modified to add the framework modulemap, removing the Header include path, and manually modifying the pyatomic_gcc #include`.

[freakboy3742]

I've just tried your SwiftWithPython project and I get the same warnings.

[frogcjn]

I have tried your project,
I just replaced the import "Python/Python.h" with @import Python; and there is no warnings on compile time.
But cannot find the module pythonimport pydevd_pycharm when running.

For my project,
did you clear the building folder? and without changing any code?

[freakboy3742]

The change in import makes no difference for me. I can't see how it could make any difference - the warnings are being reported in the compilation of the Python module, not the main.m file.

Yes, I've done a full clean every time. To confirm we're talking about the same thing - this is where I'm seeing the warnings:

As for the project not running - that's to be expected - I picked a random project I had lying around, and it looks like that project has some runtime problems. What you're seeing is expected behaviour in its current state. If you edit app/helloworld/__main__.py and remove the references to pydevd_pycharm, the app should start up.

[frogcjn]

@import Python; is for modular framework its different than #import "Python/Python.h"
So we should at same page, change it to @import Python;, at least on my Mac, this will depress all warnings.

When I use #import <Python/Python.h>, it will show warnings when build

When I use @import Python;, after cleaning build folder, it will have no warnings when build

Could you give me your Command line tool path?

[freakboy3742]

@import Python; is for modular framework its different than #import "Python/Python.h"

I know. But that's irrelevant here.

What I'm saying is that the warnings aren't being raised by main.m, so changes to main.m won't affect whether the warnings are generated. They're being raised as part of the "Prepare Build" step, not the "Build target Hello World" step.

Could you give me your Command line tool path?

My setup is the same as yours, replacing rkm for frogcjn for the default paths.

[frogcjn]

Yes. If I change it to import <Python/Python.h> it will raise on the prepare Build step, same as your warnings.

did you try @import Python with build folder clear?

I could suppress the warnings when using @import Python;, and show warnings same as yours when use import <Python/Python.h>

[freakboy3742]

Yes - I'm calling clean before every build. And I made the @import Python change a while back.

[frogcjn]

freakboy3742

You could try multiple builds after using @import Python;, It will depress warnings.

If you still want to use #import <Python/Python.h> here is the way:

python_framework_files_to_change.zip

1. change the Python.framework/Modules/module.modulemap file to:

framework module Python {
    umbrella header "Python.h"
    
    export *
    module * { export * }

    explicit module datetime {
      header "datetime.h"
      export *
    }

    explicit module errcode {
      header "errcode.h"
      export *
    }

    explicit module frameobject {
      header "frameobject.h"
      export *
    }

    explicit module marshal {
      header "marshal.h"
      export *
    }

    explicit module opcode {
      header "opcode.h"
      export *
    }

    explicit module osdefs {
      header "osdefs.h"
      export *
    }

    explicit module py_curses {
      header "py_curses.h"
      export *
    }

    explicit module pythread {
      header "pythread.h"
      export *
    }

    explicit module structmember {
      header "structmember.h"
      export *
    }

    explicit module structseq {
      header "structseq.h"
      export *
    }

    // Note: missing #include here
    exclude header "pyexpat.h"
    exclude header "pydtrace.h"
    exclude header "dynamic_annotations.h"
    
    // cpython
    exclude header "cpython/pthread_stubs.h"
    exclude header "cpython/pystats.h"
    exclude header "cpython/pyatomic_std.h"
    exclude header "cpython/pyatomic_msc.h"
    exclude header "cpython/pyatomic_gcc.h"

    // internal
    exclude header "internal/mimalloc/mimalloc.h"
    exclude header "internal/mimalloc/mimalloc/atomic.h"
    exclude header "internal/mimalloc/mimalloc/internal.h"
    exclude header "internal/mimalloc/mimalloc/prim.h"
    exclude header "internal/mimalloc/mimalloc/track.h"
    exclude header "internal/mimalloc/mimalloc/types.h"
    exclude header "internal/pycore_abstract.h"
    exclude header "internal/pycore_asdl.h"
    exclude header "internal/pycore_ast.h"
    exclude header "internal/pycore_ast_state.h"
    exclude header "internal/pycore_atexit.h"
    exclude header "internal/pycore_backoff.h"
    exclude header "internal/pycore_bitutils.h"
    exclude header "internal/pycore_blocks_output_buffer.h"
    exclude header "internal/pycore_brc.h"
    exclude header "internal/pycore_bytes_methods.h"
    exclude header "internal/pycore_bytesobject.h"
    exclude header "internal/pycore_call.h"
    exclude header "internal/pycore_capsule.h"
    exclude header "internal/pycore_cell.h"
    exclude header "internal/pycore_ceval.h"
    exclude header "internal/pycore_ceval_state.h"
    exclude header "internal/pycore_code.h"
    exclude header "internal/pycore_codecs.h"
    exclude header "internal/pycore_compile.h"
    exclude header "internal/pycore_complexobject.h"
    exclude header "internal/pycore_condvar.h"
    exclude header "internal/pycore_context.h"
    exclude header "internal/pycore_critical_section.h"
    exclude header "internal/pycore_crossinterp.h"
    exclude header "internal/pycore_descrobject.h"
    exclude header "internal/pycore_dict.h"
    exclude header "internal/pycore_dict_state.h"
    exclude header "internal/pycore_dtoa.h"
    exclude header "internal/pycore_emscripten_signal.h"
    exclude header "internal/pycore_emscripten_trampoline.h"
    exclude header "internal/pycore_exceptions.h"
    exclude header "internal/pycore_faulthandler.h"
    exclude header "internal/pycore_fileutils.h"
    exclude header "internal/pycore_fileutils_windows.h"
    exclude header "internal/pycore_floatobject.h"
    exclude header "internal/pycore_flowgraph.h"
    exclude header "internal/pycore_format.h"
    exclude header "internal/pycore_freelist.h"
    exclude header "internal/pycore_function.h"
    exclude header "internal/pycore_frame.h"
    exclude header "internal/pycore_gc.h"
    exclude header "internal/pycore_genobject.h"
    exclude header "internal/pycore_getopt.h"
    exclude header "internal/pycore_gil.h"
    exclude header "internal/pycore_global_objects.h"
    exclude header "internal/pycore_global_objects_fini_generated.h"
    exclude header "internal/pycore_global_strings.h"
    exclude header "internal/pycore_hamt.h"
    exclude header "internal/pycore_hashtable.h"
    exclude header "internal/pycore_identifier.h"
    exclude header "internal/pycore_import.h"
    exclude header "internal/pycore_importdl.h"
    exclude header "internal/pycore_initconfig.h"
    exclude header "internal/pycore_instruction_sequence.h"
    exclude header "internal/pycore_instruments.h"
    exclude header "internal/pycore_interp.h"
    exclude header "internal/pycore_intrinsics.h"
    exclude header "internal/pycore_jit.h"
    exclude header "internal/pycore_list.h"
    exclude header "internal/pycore_llist.h"
    exclude header "internal/pycore_lock.h"
    exclude header "internal/pycore_long.h"
    exclude header "internal/pycore_memoryobject.h"
    exclude header "internal/pycore_mimalloc.h"
    exclude header "internal/pycore_modsupport.h"
    exclude header "internal/pycore_moduleobject.h"
    exclude header "internal/pycore_namespace.h"
    exclude header "internal/pycore_object.h"
    exclude header "internal/pycore_object_alloc.h"
    exclude header "internal/pycore_object_stack.h"
    exclude header "internal/pycore_object_state.h"
    exclude header "internal/pycore_obmalloc.h"
    exclude header "internal/pycore_obmalloc_init.h"
    exclude header "internal/pycore_opcode_metadata.h"
    exclude header "internal/pycore_opcode_utils.h"
    exclude header "internal/pycore_optimizer.h"
    exclude header "internal/pycore_parking_lot.h"
    exclude header "internal/pycore_parser.h"
    exclude header "internal/pycore_pathconfig.h"
    exclude header "internal/pycore_pyarena.h"
    exclude header "internal/pycore_pyatomic_ft_wrappers.h"
    exclude header "internal/pycore_pybuffer.h"
    exclude header "internal/pycore_pyerrors.h"
    exclude header "internal/pycore_pyhash.h"
    exclude header "internal/pycore_pylifecycle.h"
    exclude header "internal/pycore_pymath.h"
    exclude header "internal/pycore_pymem.h"
    exclude header "internal/pycore_pymem_init.h"
    exclude header "internal/pycore_pystate.h"
    exclude header "internal/pycore_pystats.h"
    exclude header "internal/pycore_pythonrun.h"
    exclude header "internal/pycore_pythread.h"
    exclude header "internal/pycore_qsbr.h"
    exclude header "internal/pycore_range.h"
    exclude header "internal/pycore_runtime.h"
    exclude header "internal/pycore_runtime_init.h"
    exclude header "internal/pycore_runtime_init_generated.h"
    exclude header "internal/pycore_semaphore.h"
    exclude header "internal/pycore_setobject.h"
    exclude header "internal/pycore_signal.h"
    exclude header "internal/pycore_sliceobject.h"
    exclude header "internal/pycore_stackref.h"
    exclude header "internal/pycore_strhex.h"
    exclude header "internal/pycore_structseq.h"
    exclude header "internal/pycore_symtable.h"
    exclude header "internal/pycore_sysmodule.h"
    exclude header "internal/pycore_time.h"
    exclude header "internal/pycore_token.h"
    exclude header "internal/pycore_traceback.h"
    exclude header "internal/pycore_tracemalloc.h"
    exclude header "internal/pycore_tstate.h"
    exclude header "internal/pycore_tuple.h"
    exclude header "internal/pycore_typeobject.h"
    exclude header "internal/pycore_typevarobject.h"
    exclude header "internal/pycore_ucnhash.h"
    exclude header "internal/pycore_unicodeobject.h"
    exclude header "internal/pycore_unicodeobject_generated.h"
    exclude header "internal/pycore_unionobject.h"
    exclude header "internal/pycore_uop_ids.h"
    exclude header "internal/pycore_uop_metadata.h"
    exclude header "internal/pycore_warnings.h"
    exclude header "internal/pycore_weakref.h"
}

2. replace Python.framework/Versions/3.13/include/python3.13/cpython/pyatomic.h, in the file:
   • replace cpython/pyatomic_gcc.h with pyatomic_gcc.h
   • replace cpython/pyatomic_std.h with pyatomic_std.h
   • replace cpython/pyatomic_msc.h with pyatomic_msc.h

3. Edit Python.framework/Versions/3.13/include/python3.13/py_curses.h
   move this part

   #if defined(WINDOW_HAS_FLAGS) && defined(__APPLE__)
   /* gh-109617, gh-115383: we can rely on the default value for NCURSES_OPAQUE on
      most platforms, but not on macOS. This is because, starting with Xcode 15,
      Apple-provided ncurses.h comes from ncurses 6 (which defaults to opaque
      structs) but can still be linked to older versions of ncurses dynamic
      libraries which don't provide functions such as is_pad() to deal with opaque
      structs. Setting NCURSES_OPAQUE to 0 is harmless in all ncurses releases to
      this date (provided that a thread-safe implementation is not required), but
      this might change in the future. This fix might become irrelevant once
      support for macOS 13 or earlier is dropped. */
   #define NCURSES_OPAQUE 0
   #endif
   

   after

   #if defined(HAVE_NCURSESW_NCURSES_H)
   #  include <ncursesw/ncurses.h>
   #elif defined(HAVE_NCURSESW_CURSES_H)
   #  include <ncursesw/curses.h>
   #elif defined(HAVE_NCURSES_NCURSES_H)
   #  include <ncurses/ncurses.h>
   #elif defined(HAVE_NCURSES_CURSES_H)
   #  include <ncurses/curses.h>
   #elif defined(HAVE_NCURSES_H)
   #  include <ncurses.h>
   #elif defined(HAVE_CURSES_H)
   #  include <curses.h>
   #endif
   

   and wrap #ifndef around #define NCURSES_OPAQUE 0

   #ifndef NCURSES_OPAQUE
   #define NCURSES_OPAQUE 0
   #endif