Standard Library Header Units and Modules - tracking issue

[StephanTLavavej]

📜 Documentation

• P1502R1 Standard Library Header Units
  • P1502R1 Standard Library Header Units #60
  • Walkthrough: Build and import header units in Microsoft Visual C++
  • Walkthrough: Import STL libraries as header units
• P2465R3 Standard Library Modules std and std.compat
  • P2465R3 Standard Library Modules std And std.compat #2930
  • Tutorial: Import the C++ standard library using modules from the command line

🚧 Status

• Standard Library Header Units are ready for production use, for MSVC only.
  • Shipped in VS 2019 16.11.
• Standard Library Modules are ready for production use, for MSVC only.
  • Shipped in VS 2022 17.5: P2465R3 Standard Library Modules std And std.compat #3108
• The C++23 named modules std and std.compat are available when compiling in C++20 mode.
  • Shipped in VS 2022 17.8: Support the Standard Library Modules in C++20 mode #3977
  • This extension is supported by the major Standard Library implementations via an informal agreement between implementers.
• #include <meow> can coexist with import std; in the same translation unit, in that order.
  • Shipped in VS 2022 17.10: Add extern "C++" as a temporary workaround for #include/import coexistence #4154
  • The other order, import std; before #include <meow>, will still cause compiler errors. We're working on a long-term solution.

We've reported the following compiler bugs; this issue is for tracking the status of these compiler bugs and removing workarounds/adding test coverage as compiler fixes are released. As you can see below, a ton of compiler bugs have been fixed. If you're using header units or named modules, I strongly recommend using the latest version of VS 2022.

As /clr C++20 support is a work in progress, we are not yet testing Standard Library Header Units or Modules under /clr.

❌ Clang

• Clang currently lists C++20 modules as not yet fully implemented: https://clang.llvm.org/cxx_status.html#cxx20
• I've heard from users that Clang 19 works with MSVC's Standard Library Modules but I haven't had a chance to test this yet.

🐛 EDG (IntelliSense Front-End)

• EDG is currently investigating test failures. STL test coverage is not yet enabled. See Report EDG bugs #1621.

🪲 Active, UCRT

• OS-33790456 "static inline functions cannot be called when using modules"
  • Bug bash report: <ctime>: time()/localtime()/mktime() emit error C2129: static function declared but not defined StephanTLavavej/STL#21

🐞 Active, C1XX (MSVC Front-End)

• <filesystem>: compiler error using directory_iterator/recursive_directory_iterator inside a struct/class with modules #3661
• DevCom-1256361 VSO-1248260 "[C++][Modules][GMF] Named module leaks names from GMF"
  • This will cause the std module to export too much stuff.
• VSO-1236041 "Standard Library Header Units: bogus error LNK2019: unresolved external symbol pair(piecewise_construct_t, tuple, tuple)"
  • Requires an IFC update
• VSO-1953157 "Modules: MSVC should emit an error when an exported declaration follows a non-exported declaration"
• DevCom-10729775 VSO-2225134 "Standard Library Modules: Random number machinery emits bogus error C2512: 'std::_Unsigned128': no appropriate default constructor available"
  • Originally Standard Library Modules: uniform_real_distribution emits error C2512: 'std::_Unsigned128': no appropriate default constructor available #4899
• VSO-2226569 "Standard Library Modules: generate_canonical causes C1XX ICE: find_pending_comparison_function_for_definition(function) == std::end(pending_comparison_functions_for_definition)"
• VSO-2227713 "Modules: Exported function templates can't use std::cout properly"
• <thread>: Using this_thread::sleep_for in a module results in compilation errors #5203
  • Need to analyze.
• DevCom-10736790 VSO-2238628 "Reference NTTPs broken in modules (ICE)"
  • Reported to us as <variant>: Failed to use an empty class as template parameter #5254

🛠️ Upcoming Improvements (C1XX)

• Modules bugs are constantly being fixed, but nothing I'm tracking is queued up for release at the moment.

❔ Possible Improvements (C1XX/MSBuild)

• VSO-1330581 "Throughput for Standard Library Header Units"
• VSO-1469758 "Standard Library Header Units: Possible IFC size reductions?"
• VSO-1568412 "Standard Library Modules: Enhanced diagnostics for needing import std.compat;, std::printf, or std::cout"
• VSO-1621267 "Standard Library Modules: Enhanced diagnostics for missing macros"
• VSO-1922468 "Standard Library Modules: Helpful diagnostic when import std; is seen, but std.ifc/std.obj need to be built"
• VSO-1922469 "Support C++20 builds of import std; and import std.compat;"
• VSO-1987212 "[C++ Modules DT perf] STL modules are always scanned in designtime regardless of their usage in code"

🩹 Permanent Workarounds

• VSO-1308657 "Standard Library Header Units: std::projected::operator*() error LNK2019: unresolved external symbol"
  • Perma-workaround: provide a definition that calls abort().

😻 Fixed, Workaround Removed

• Fixed in 16.8:
  • DevCom-1160043 VSO-1168062 "Standard Library Header Units: Bogus error C2131 when using compare functions at compile-time"
  • DevCom-1160145 VSO-1180193 "Standard Library Header Units: error LNK2019 'unresolved external symbol' when a function is declared in one header and defined in another"
• Fixed in 16.9:
  • DevCom-1159869 VSO-1180127 "Standard Library Header Units: Bogus warning C4373 about virtual function overriding"
  • DevCom-1159995 VSO-1180128 "Standard Library Header Units: UDLs don't compile, depending on the order of imports"
  • DevCom-1160260 VSO-1180207 "Standard Library Header Units: Bogus error C2752: 'std::pointer_traits<_Voidptr>': more than one partial specialization matches the template argument list"
  • DevCom-1161187 VSO-1181303 "Standard Library Header Units: Bogus error C2248 'cannot access private member'"
  • DevCom-1162644 VSO-1181305 "Standard Library Header Units: __pragma(warning(disable : 4996)) doesn't take effect"
  • DevCom-1224512 VSO-1236022 "Standard Library Header Units: Bogus error C2027: use of undefined type 'std::char_traits'"
  • VSO-1236034 "Standard Library Header Units: bogus error LNK2005: std::_Yarn<char>::operator=(char const *) already defined"
  • VSO-1236047 "Standard Library Header Units: std::optional, c1xx!FindConversionFunctions() assertion failed: PF_ISSET(ParsingClassTemplateDefn)"
  • VSO-1237804 "Standard Library Header Units: std::variant, c1xx!Module::InterfaceReader::materialize_function() assertion failed"
  • VSO-1237145 "Standard Library Header Units: bogus error C7599: 'std::ranges::operator ==': a trailing requires clause is only allowed on a templated function"
• Fixed and implemented in 16.10:
  • VSO-1271718 "Standard Library Header Units ICE with C++20 chrono"
  • VSO-1180134 DevCom-1160041 "Standard Library Header Units: deprecated attributes aren't imported"
  • VSO-1273005 "[Modules] c1xx does not fully resolve templates imported from IFC in the context of trailing requirements"
  • VSO-1284279 "Standard Library Header Units: memory_resource ICE: Mapping already has an entry defined at this slot"
  • VSO-1287222 "[modules] transitive header unit imports do not import macros"
  • VSO-1287925 "/headerUnit:angle ICEs"
  • VSO-1307828 "Standard Library Header Units: ICE with queue's friend bool operator=="
  • VSO-1309454 "Standard Library Header Units: CTAD fails when deduction guides are duplicated across header units"
  • VSO-1314139 "Cannot build header units for STL headers with 'Translate Includes to Imports'"
  • VSO-1232145 "EDG ICEs when consuming Standard Library Header Units"
  • Implemented /headerUnit:angle vector=vector.ifc
  • Implemented support for header-units.json via /sourceDependencies:directives and /translateInclude
  • Implemented /exportHeader /headerName:angle algorithm type_traits vector
• Fixed in 16.11:
  • VSO-1330589 "Standard Library Header Units: Bogus error C2131 with _Nontrivial_dummy_type's constructor"
  • VSO-1330591 "Standard Library Header Units: Bogus error C2676 with type_info's equality operator"
  • DevCom-1440183 VSO-1337327 "Constexpr Function Involving String Does Not Compile When Consuming Modules"
• Fixed in 17.0:
  • <format>: error C2668 when import <format> as header unit and use std::format_to #1919 VSO-1329224 "Bogus error C2668 when importing <format> as a header unit"
  • VSO-1329976 "Standard Library Header Units: _CrtGetAllocHook emits warning C5106: macro redefined with different parameter names"
• Fixed in 17.1:
  • DevCom-1511903 VSO-1384883 "Standard Library Header Units: Bogus error C2440 when convertible_to constrains a constructor"
  • VSO-1409853 "Standard Library Header Units: <ranges> compiler assertion: previous_element == tokenInputStack.TopOfStack(), aliastemplates.cpp 1010"
  • VSO-1433873 "Standard Library Header Units: Adding template <int = 0> to vformat() emits warnings C4265 and C4365"
• Fixed in 17.2:
  • VSO-1464637 "Standard Library Header Units: #pragma warning doesn't always suppress warnings in templates"
  • VSO-1466711 "/scanDependencies generates incorrect "logical-name" in JSON output"
  • VSO-1471374 "Standard Library Header Units: Deduplication emits fatal error C1116: unrecoverable error importing module, with <concepts>"
  • VSO-1471376 "Standard Library Header Units: Confusing diagnostic for fatal error C1116: unrecoverable error importing module ''"
  • VSO-1471382 "Standard Library Header Units: Deduplication emits error C2672: 'count_if': no matching overloaded function found"
• Fixed in 17.3:
  • VSO-1496074 "Standard Library Modules: C++20 chrono::days emits bogus error C2131"
  • VSO-1496084 "Standard Library Header Units: Splitting <chrono> emits fatal error C1116: unrecoverable error importing header unit '<chrono>'"
  • VSO-1496493 "Standard Library Modules: Implementing std.ixx emits fatal error C1011: cannot locate standard module interface"
  • VSO-1497379 "Standard Library Modules: extern "C" function declarations taking extern "C" types emit bogus error LNK2019: unresolved external symbol"
  • VSO-1504872 "Standard Library Modules: extern "C" function mentioning entity with module linkage is being marked with module linkage"
  • VSO-1526632 "Standard Library Header Units: Hidden friend operators cause ICE in Module::InterfaceReader::materialize_function<Module::helpers::MemberBuilder>()"
  • VSO-1513874 "Changing construct_at SFINAE causes Modules bogus error C2660: 'operator new': function does not take 3 arguments"
  • VSO-1533852 "Standard Library Modules: The Mega-Bug"
    • <codecvt>: fatal error C1001: Internal compiler error.
    • <any>: error C2679: binary '!=': no operator found which takes a right-hand operand of type 'const type_info' (or there is no acceptable conversion)
    • <memory>: error C2679: binary '==': no operator found which takes a right-hand operand of type 'const type_info' (or there is no acceptable conversion)
    • <functional>: error C2027: use of undefined type 'type_info'
    • <typeinfo>: error C2872: 'type_info': ambiguous symbol
    • <typeindex>: error C2440: 'initializing': cannot convert from 'initializer list' to 'std::type_index'
    • <regex>: Assertion failed: Nerrors > 0, file D:\msvc\src\vctools\Compiler\CxxFE\sl\p1\c\trees.c
  • VSO-1540220 "Standard Library Modules: /d1ifcInlineFunctions asserts (bits::rep(typeEncoding.basicEncoding_) & 0x0000000F) != 0 for printf(_Format, ...)"
  • VSO-1543660 "Standard Library Header Units: <expected> ICEs with Assertion failed: IsInClassDefn()"
  • VSO-1541639 "Standard Library Modules: error C3083: '_Iterator_base12': the symbol to the left of a '::' must be a type"
  • VSO-1509503 "Standard Library Header Units: Deduplication interferes with ADL for global operator&"
• Fixed in 17.4:
  • VSO-1538668 "Compiler does not bind names inside of lambda nested in function template with parameter pack"
  • <ranges>: could not use std::views::istream StephanTLavavej/STL#13 VSO-1581022 "Standard Library Modules: inline constexpr variables for ranges::views emit error LNK2019: unresolved external symbol"
  • VSO-1567850 "Standard Library Modules: UCRT headers emit error C2294: cannot export symbol 'memcpy_s' because it has internal linkage"
  • VSO-1538666 "Compiler does not bind NSDMIs inside class templates"
  • VSO-1522454 "Standard Library Modules: Unresolved external symbol when extern "C++" is directly applied to a class with a static member function"
  • <ranges>: ICE with std::views::join_with StephanTLavavej/STL#17 VSO-1582910 "Standard Library Modules: ICE with std::views::join_with"
  • <variant>: Constructor inheritance causes ICE when class in named module inherits from class imported from module std StephanTLavavej/STL#18 VSO-1582916 "Standard Library Modules: ICE when class in named module inherits from variant with inheriting constructors"
  • VSO-1579484 "Standard Library Header Units: charconv refactoring + topo sort = error LNK2019: unresolved external symbol"
  • VSO-1593115 "Standard Library Modules: ICE with chunk_view of input iterators"
• Fixed in 17.5:
  • VSO-1611408 "Standard Library Modules: Compiler assertion failed: inserted, file template.cpp, line 10275"
  • VSO-1611411 "Standard Library Modules: Compiler assertion failed: IsInClassDefn() && (region->owner == SU_TAG), file reader.cpp, line 2592"
  • <fstream>: Importing as a header unit leads to error C2079: undefined class 'std::basic_ofstream<char,std::char_traits<char>>' #3112 DevCom-10154236 "<fstream>: Importing as a header unit leads to error C2079: undefined class 'std::basic_ofstream<char,std::char_traits<char>>'"
  • VSO-1592399 "Standard Library Modules: destroy_at() emits bogus error C2131: expression did not evaluate to a constant"
  • VSO-1592420 "Standard Library Modules: iota_view emits bogus error C7608: atomic constraint should be a constant expression"
  • VSO-1593015 "Standard Library Modules: constexpr vector::resize() ICE"
• Fixed in 17.6:
  • <ranges>: Usage of ranges::iterator_t causes ICE StephanTLavavej/STL#23 VSO-1584629 "Standard Library Modules: Usage of ranges::iterator_t causes ICE"
  • <atomic>: Building a header unit with /ZI emits warning C5260 #3287 VSO-1706387 "<atomic>: Building a header unit with /ZI emits warning C5260"
  • VSO-1692469 "Standard library header units: deduplication causes ICE when combined with variable template specialization"
  • VSO-1592409 "Standard Library Modules: source_location::column() misbehaves in user headers"
  • std::filesystem::directory_iterator code compiles OK with <filesytem> fails with std module, 17.5 Preview 2  #3330 DevCom-10247495 VSO-1714748 "MSVC forces us to export some internal enumeration functions for working with C++ modules"
  • Standard Library Modules: bogus error C2504: 'std::filesystem::_Dir_enum_impl': base class undefined #3333 VSO-1715231 "Standard Library Modules: Bogus error C2504: 'std::filesystem::_Dir_enum_impl': base class undefined"
  • DevCom-10241527 VSO-1711320 "VS 2022 17.5.0 Preview 2 import header unit charconv gives wrong result"
  • VSO-1592407 "Standard Library Modules: using namespace std::chrono; for UDLs emits bogus error C3688: invalid literal suffix"
  • VSO-1705654 "Standard Library Modules: /analyze ICE"
  • VSO-1593163 "Standard Library Modules: osyncstream move assignment emits bogus error C3083: 'basic_ostream<char,std::char_traits<char> >': the symbol to the left of a '::' must be a type"
  • VSO-1592329 "Standard Library Modules: Parallel sort() emits bogus error C2065: '_Atomic_counter_t': undeclared identifier"
  • <format> + <chrono>: Could not format durations StephanTLavavej/STL#14 VSO-1582358 "Standard Library Modules: chrono formatting emits bogus error C3861: '_Fill_tm': identifier not found"
  • <chrono>: Spurious warning C4702: unreachable code StephanTLavavej/STL#15 VSO-1582381 "Standard Library Modules: Bogus warning C4702: unreachable code"
  • VSO-1720644 "Standard Library Modules: Scanning dependencies emits bogus warning C5244: '#include <meow>' in the purview of module 'std' appears erroneous"
  • VSO-1735035 "Standard Library Modules: ICE with P2165R4's tuple changes"
  • DevCom-10097021 VSO-1581774 "C++ STL header <format> causes C4296 warning that need to be suppressed every time" (with header units)
  • VSO-1752496 "Standard Library Modules: Suppressing C4996 deprecation warnings doesn't work with static constexpr data members"
• Fixed in 17.7:
  • DevCom-10283892 VSO-1749846 "C++ modules std::expected error C2280 attempting to reference a deleted function"
  • DevCom-10404496 "[C++][Modules] Using std::views::iota in member function in module causes ICE"
• Fixed in 17.8:
  • DevCom-10404498 VSO-1843798 "[C++][Modules] Using std::views::repeat in member functions gives bogus error about lack of compare header"
  • DevCom-10324326 VSO-1780122 "'...': there are no parameter packs available to expand module"
  • VSO-1475786 "Standard Library Header Units: Deduplication emits bogus errors related to numeric_limits"
    • Fixed at some point between VS 2022 17.1 and 17.8.
  • VSO-1593161 "Standard Library Modules: ctype<char>::table_size should be a compile-time constant"
• Fixed in 17.9:
  • DevCom-10359857 VSO-1817670 "[C++][Modules] std::mdspan is unusable with C++20 modules"
  • DevCom-10452770 VSO-1877720 "Bug with using modules and getting error "multiple defined std::basic_string_view symbols""
  • DevCom-1359160 VSO-1291691 "Compilation fails when both C++ Modules and Address Sanitizer are present"
• Fixed in 17.10:
  • VSO-1592395 "Standard Library Modules: <chrono> year_month_weekday and year_month_weekday_last emit bogus error C2131: expression did not evaluate to a constant"
  • VSO-1555263 "Standard Library Header Units: Local array with day{31} initializers emits 'not yet implemented' internal compiler error"
  • VSO-1913621 "extern "C++" doesn't avoid all problems when mixing includes and imports"
  • VSO-1914077 "Modules: extern "C++" interferes with ADL"
  • DevCom-10452704 VSO-1880921 "[Feedback][std:c++latest] C++ compile error when both including a header and calling std::filesystem::path::generic_string() in a module unit"
  • DevCom-10542001 VSO-1930627 "error C1116 unrecoverable error importing module 'std' when constructing and exporting a std::chrono::zoned_time object"
    • Duplicate: VSO-1951431 "Standard Library Modules: chrono::time_point formatting emits bogus error C3861: '_Fill_tm': identifier not found"
    • Duplicate: VSO-1951433 "Standard Library Modules: utc_clock::now() emits fatal error C1116: unrecoverable error importing module 'std'."
  • VSO-1538698 "Better handling for non-exported friend function declarations"
  • VSO-1975579 "Standard Library Modules: fatal error C1116: unrecoverable error importing module 'std'. Specialization of 'std::invoke_result_t' with arguments '_Fn, _Ty...'"
  • DevCom-10519854 VSO-1919553 "Using std::views::zip with std::views::iota in C++20 module results in C1001 Internal compiler error"
• Fixed in 17.12:
  • DevCom-10506520 VSO-1912515 "Partial specialization of std::formatter in module is not found in other module."
  • Standard Library Modules: error C2039: 'promise_type': is not a member of 'std::coroutine_traits<TestModule::FireAndForget>' #4826
  • DevCom-10652168 VSO-2053126 "C++ modules: Internal compiler error when using std::stacktrace"
  • DevCom-10313766 VSO-1775715 "Using std::format in a module requires including <format> header in .cpp files using that module"
• Fixed in 17.13:
  • VSO-2293247 "/Zc:preprocessor does not terminate macro definitions properly"
    • This affected /scanDependencies for header units
• Fixed in 17.14 Preview 1:
  • DevCom-10755613 VSO-2267438 "A custom formatter and a module raise errors in __msvc_formatter.hpp"

🏗️ Building Deduplicated Header Units

Here's the part of our Python-powered test harness that builds "deduplicated" header units by topologically sorting them:

STL/tests/std/tests/P1502R1_standard_library_header_units/custom_format.py

Lines 56 to 109 in ef62d3f

	# Compiler options, common to both scanning dependencies and building header units.
	clOptions = ['/exportHeader', '/headerName:angle', '/translateInclude', '/Fo', '/MP']
	# Store the list of headers to build.
	allHeaders = getAllHeaders(os.path.join(litConfig.cxx_headers, 'header-units.json'))
	# Generate JSON files that record how these headers depend on one another.
	if noisyProgress:
	print('Scanning dependencies...')
	cmd = [test.cxx, *test.flags, *test.compileFlags, *clOptions, '/scanDependencies', '.\\', *allHeaders]
	yield TestStep(cmd, shared.execDir, shared.env, False)
	# The JSON files also record what object files will be produced.
	# IFC filenames and OBJ filenames follow different patterns. For example:
	# <filesystem> produces filesystem.ifc and filesystem.obj
	# <xbit_ops.h> produces xbit_ops.h.ifc and xbit_ops.obj
	# We can easily synthesize IFC filenames, but it's easier to get the OBJ filenames from the JSON files.
	# This dictionary powers the topological sort.
	# Key: Header name, e.g. 'bitset'.
	# Value: List of dependencies that remain to be built, e.g. ['iosfwd', 'limits', 'xstring'].
	remainingDependencies = {}
	# Read the JSON files, storing the results in objFilenames and remainingDependencies.
	for hdr in allHeaders:
	with open(os.path.join(outputDir, f'{hdr}.module.json')) as file:
	jsonObject = json.load(file)
	objFilenames.append(jsonObject['rules'][0]['primary-output'])
	remainingDependencies[hdr] = [req['logical-name'] for req in jsonObject['rules'][0]['requires']]
	# Build header units in topologically sorted order.
	while len(remainingDependencies) > 0:
	# When a header has no remaining dependencies, it is ready to be built.
	readyToBuild = [hdr for hdr, dep in remainingDependencies.items() if len(dep) == 0]
	# Each layer can be built in parallel.
	if noisyProgress:
	print('Building deduplicated header units:', ' '.join(readyToBuild))
	cmd = [test.cxx, *test.flags, *test.compileFlags, *clOptions, *consumeBuiltHeaderUnits, *readyToBuild]
	yield TestStep(cmd, shared.execDir, shared.env, False)
	# Update remainingDependencies by doing two things.
	# hdr, dep is the current key-value pair.
	# First, keep `if len(dep) > 0`. (Otherwise, we just built hdr.)
	# Second, filter dep, eliminating anything that appears in readyToBuild. (If we're left with
	# an empty list, then hdr will be ready to build in the next iteration.)
	remainingDependencies = {
	hdr: [d for d in dep if d not in readyToBuild]
	for hdr, dep in remainingDependencies.items() if len(dep) > 0
	}
	# Add compiler options to consume the header units that we just built.
	for hdr in readyToBuild:
	consumeBuiltHeaderUnits += ['/headerUnit:angle', f'{hdr}={hdr}.ifc']

With a bit of work, this can be extracted into a standalone Python script. (With significantly more work, it could be converted into CMake.)