Newlib nano variant footprint reduction

[stephanosio]

Problem description

As part of the Zephyr SDK 0.14.2 release, the C99 format specifier support was deliberately enabled for the newlib "nano" variant in spite of the significant footprint increase, in order to fix the bug #45336 (C99 format specifiers are not supported by the "nano" variant of the newlib C library).

The rationale given for this change was as follows (see zephyrproject-rtos/sdk-ng#469 (comment)):

this is a necessary change for us to be able to universally make use of the C99 format specifiers in the Zephyr codebase, which is necessary for C99 compliance as well as writing and supporting portable code.

As noted in #45336, with the introduction of the picolibc as a replacement for the minimal libc, the lowest end requirement of the newlib nano C library can be loosened

the newlib nano variant footprint is still considerably smaller than that of the newlib full variant even with the C99 format specifier support enabled, so the effectiveness of the nano variant is by no means nullified by this change.

While reasonable, the picolibc has not been adopted by many because it is relatively (very) new, and this left the existing users of the newlib "nano" variant who have switched to using the Zephyr SDK 0.14.2 release with a non-negligible footprint increase in their binaries.

Proposed change

Until the picolibc becomes more widely adopted by the existing users of the newlib "nano" variant, keep the C99 format specifier support disabled for the newlib "nano" variant at the cost of breaking the C99 standard compatibility, in order to ease the transition to the picolibc.

The following changes are to be implemented:

1. Revert the Zephyr SDK patch enabling the C99 format specifier support for the newlib "nano" variant.
2. Document that the C99 format specifiers are not supported when using the newlib "nano" variant.
3. Make Zephyr always use the newlib "full" variant when CONFIG_NEWLIB_LIBC=y, unless CONFIG_NEWLIB_LIBC_NANO is explicitly selected by the user (i.e. enable the newlib "nano" variant if any only if CONFIG_NEWLIB_LIBC_NANO=y).

Regarding no. 3, the newlib "nano" variant is currently enabled by default when CONFIG_NEWLIB_LIBC=y and the selected toolchain-architecture includes the newlib "nano" variant support, even if CONFIG_NEWLIB_LIBC_NANO is not selected by the user:

zephyr/lib/libc/Kconfig

Lines 81 to 83 in fd07675

	config NEWLIB_LIBC_NANO
	bool "Build with newlib-nano C library"
	depends on HAS_NEWLIB_LIBC_NANO

config TOOLCHAIN_ZEPHYR_0_14
	def_bool y
	select HAS_NEWLIB_LIBC_NANO if (ARC || (ARM && !ARM64) || RISCV)

This means that the newlib "nano" variant will be used when CONFIG_NEWLIB_LIBC=y (and CONFIG_NEWLIB_LIBC_NANO is not explicitly specified) for the ARC, ARM and RISC-V targets while the newlib "full" variant will be used for the rest.

This is ambiguous and especially problematic when there exists an incompatibility between the "full" and the "nano" variants of the newlib; therefore, it is necessary to make the selection of the newlib "nano" variant explicit such that it is only selected when the user specifically requests for it by setting CONFIG_NEWLIB_LIBC_NANO=y.

Concerns and Unresolved Questions

By implementing the proposed change, there will exist a functional difference (incompatibility) between the "full" and "nano" variants of the newlib C library in which the latter does not support the size modifiers introduced in the C99 standard such as hh (aka. "C99 format specifiers"), which may lead to even more complications and confusions.

Alternatives

Keep the C99 format specifier support enabled for the newlib "nano" variant and require all highly footprint-sensitive users to migrate to using the picolibc (it should be as simple as replacing CONFIG_NEWLIB_LIBC=y with CONFIG_PICOLIBC=y in their configs).

Additional context

Related discussion on Discord: https://discord.com/channels/720317445772017664/883404732423606303/991694926272278558

Implementation Status

• Revert "newlib: Enable C99 format specifier support for newlib-nano" sdk-ng#525
• Document that the C99 format specifiers are not supported when using the newlib "nano" variant: doc: languages: c: Add formatted output section #50462
• Make Zephyr always use the newlib "full" variant when CONFIG_NEWLIB_LIBC=y, unless CONFIG_NEWLIB_LIBC_NANO is explicitly selected by the user (i.e. enable the newlib "nano" variant if any only if CONFIG_NEWLIB_LIBC_NANO=y).

[stephanosio]

cc @andyross @carlescufi @henrikbrixandersen @keith-packard

[keith-packard]

(hrm, I thought I commented on this, but it doesn't appear here?).

What parts of C99 does Zephyr want to use? I suspect the 'expensive' pieces in newlib are related to thousands separators and multi-byte character support, neither of which are likely relevant in a Zephyr context. As a comparison, I added 'thousands separator' support in picolibc by simply ignoring the formatting code -- the C locale specifies an empty string for the separator, so that was sufficient. Newlib, as it must retain API compatibility for Cygwin, has a pile of complicated locale support code that is unusable due to the lack of actual locale infrastructure for anything but Cygwin...

[andyross]

That's a good point, limiting this to "CONFIG_LIBC_LOCALE_SUPPORT" might be an easier kconfig for users to understand than "C99 format specifiers", plausibly permit yet more aggressive feature removal in other areas, and extend nicely to pico/minimal libcs in a natural way.

But broadly: yes, all of that sounds great to me. In Zephyr's domain things like "exhaustive standards compliance" and "system footprint" are always going to be in contention and we're going to need dials to manage the tradeoffs.

[stephanosio]

To be voted in the TSC in order to make sure that everyone is in agreement.

[stephanosio]

No objections raised in the TSC.

The motion to "keep the C99 format specifier support disabled for the newlib 'nano' variant at the cost of breaking the C99 standard compatibility" passed.

I will implement this change.