Add thread-safe locale handling

This (large) commit allows locales to be used in threaded perls on
platforms that support it.  This includes recent Windows and Posix 2008
ones.

Author: khwilliamson

dist/ExtUtils-ParseXS/lib/perlxs.pod

@@ -2195,7 +2195,7 @@ To summarize, here's what to expect and how to handle locales in XS code:
 =item Non-locale-aware XS code
 
 Keep in mind that even if you think your code is not locale-aware, it
-may call a C library function that is.  Hopefully the man page for such
+may call a library function that is.  Hopefully the man page for such
 a function will indicate that dependency, but the documentation is
 imperfect.
 
@@ -2231,9 +2231,107 @@ L<perlapi/STORE_LC_NUMERIC_FORCE_TO_UNDERLYING>, and
 L<perlapi/RESTORE_LC_NUMERIC> should be used to affect any needed
 change.
 
-However, some alien libraries that may be called do set it, such as
-C<Gtk>.  This can cause problems for the perl core and other modules.
-Starting in v5.20.1, calling the function
+But, starting with Perl v5.28, locales are thread-safe on platforms that
+support this functionality.  Windows has this starting with Visual
+Studio 2005.  Many other modern platforms support the thread-safe POSIX
+2008 functions.  The C C<#define> C<USE_THREAD_SAFE_LOCALE> will be
+defined iff this build is using these.  From Perl-space, the read-only
+variable C<${SAFE_LOCALES}> is 1 if either the build is not threaded, or
+if C<USE_THREAD_SAFE_LOCALE> is defined; otherwise it is 0.
+
+The way this works under-the-hood is that every thread has a choice of
+using a locale specific to it (this is the Windows and POSIX 2008
+functionality), or the global locale that is accessible to all threads
+(this is the functionality that has always been there).  The
+implementations for Windows and POSIX are completely different.  On
+Windows, the runtime can be set up so that the standard
+L<C<setlocale(3)>> function either only knows about the global locale or
+the locale for this thread.  On POSIX, C<setlocale> always deals with
+the global locale, and other functions have been created to handle
+per-thread locales.  Perl makes this transparent to perl-space code.  It
+continues to use C<POSIX::setlocale()>, and the interpreter translates
+that into the per-thread functions.
+
+All other locale-senstive functions automatically use the per-thread
+locale, if that is turned on, and failing that, the global locale.  Thus
+calls to C<setlocale> are ineffective on POSIX systems for the current
+thread if that thread is using a per-thread locale.  If perl is compiled
+for single-thread operation, it does not use the per-thread functions,
+so C<setlocale> does work as expected.
+
+If you have loaded the L<C<POSIX>> module you can use the methods given
+in L<perlcall> to call L<C<POSIX::setlocale>|POSIX/setlocale> to safely
+change or query the locale (on systems where it is safe to do so), or
+you can use the new 5.28 function L<perlapi/Perl_setlocale> instead,
+which is a drop-in replacement for the system L<C<setlocale(3)>>, and
+handles single-threaded and multi-threaded applications transparently.
+
+There are some locale-related library calls that still aren't
+thread-safe because they return data in a buffer global to all threads.
+In the past, these didn't matter as locales weren't thread-safe at all.
+But now you have to be aware of them in case your module is called in a
+multi-threaded application.  The known ones are
+
+ asctime()
+ ctime()
+ gcvt() [POSIX.1-2001 only (function removed in POSIX.1-2008)]
+ getdate()
+ wcrtomb() if its final argument is NULL
+ wcsrtombs() if its final argument is NULL
+ wcstombs()
+ wctomb()
+
+Some of these shouldn't really be called in a Perl application, and for
+others there are thread-safe versions of these already implemented:
+
+ asctime_r()
+ ctime_r()
+ Perl_langinfo()
+
+The C<_r> forms are automatically used, starting in Perl 5.28, if you
+compile your code, with
+
+ #define PERL_REENTRANT
+
+See also L<perlapi/Perl_langinfo>.
+You can use the methods given in L<perlcall>, to get the best available
+locale-safe versions of these
+
+ POSIX::localeconv()
+ POSIX::wcstombs()
+ POSIX::wctomb()
+
+And note, that some items returned by C<Localeconv> are available
+through L<perlapi/Perl_langinfo>.
+
+The others shouldn't be used in a threaded application.
+
+Some modules may call a non-perl library that is locale-aware.  This is
+fine as long as it doesn't try to query or change the locale using the
+system C<setlocale>.  But if these do call the system C<setlocale>,
+those calls may be ineffective.  Instead,
+L<C<Perl_setlocale>|perlapi/Perl_setlocale> works in all circumstances.
+Plain setlocale is ineffective on multi-threaded POSIX 2008 systems.  It
+operates only on the global locale, whereas each thread has its own
+locale, paying no attention to the global one.  Since converting
+these non-Perl libraries to C<Perl_setlocale> is out of the question,
+there is a new function in v5.28
+C<switch_to_global_locale> that will
+switch the thread it is called from so that any system C<setlocale>
+calls will have their desired effect.  The function
+L<C<sync_locale>|perlapi/sync_locale> must be called before returning to
+perl.
+
+This thread can change the locale all it wants and it won't affect any
+other thread, except any that also have been switched to the global
+locale.  This means that a multi-threaded application can have a single
+thread using an alien library without a problem; but no more than a
+single thread can be so-occupied.  Bad results likely will happen.
+
+In perls without multi-thread locale support, some alien libraries,
+such as C<Gtk> change locales.  This can cause problems for the Perl
+core and other modules.  For these, before control is returned to
+perl, starting in v5.20.1, calling the function
 L<sync_locale()|perlapi/sync_locale> from XS should be sufficient to
 avoid most of these problems.  Prior to this, you need a pure Perl
 statement that does this:

dist/threads/lib/threads.pm

@@ -5,7 +5,7 @@ use 5.008;
 use strict;
 use warnings;
 
-our $VERSION = '2.21';      # remember to update version in POD!
+our $VERSION = '2.22';      # remember to update version in POD!
 my $XS_VERSION = $VERSION;
 $VERSION = eval $VERSION;
 
@@ -937,6 +937,33 @@ C<chdir()>) will affect all the threads in the application.
 On MSWin32, each thread maintains its own the current working directory
 setting.
 
+=item Locales
+
+Prior to Perl 5.28, locales could not be used with threads, due to various
+race conditions.  Starting in that release, on systems that implement
+thread-safe locale functions, threads can be used, with some caveats.
+This includes Windows starting with Visual Studio 2005, and systems compatible
+with POSIX 2008.  See L<perllocale/Multi-threaded operation>.
+
+Each thread (except the main thread) is started using the C locale.  The main
+thread is started like all other Perl programs; see L<perllocale/ENVIRONMENT>.
+You can switch locales in any thread as often as you like.
+
+If you want to inherit the parent thread's locale, you can, in the parent, set
+a variable like so:
+
+    $foo = POSIX::setlocale(LC_ALL, NULL);
+
+and then pass to threads->create() a sub that closes over C<$foo>.  Then, in
+the child, you say
+
+    POSIX::setlocale(LC_ALL, $foo);
+
+Or you can use the facilities in L<threads::shared> to pass C<$foo>;
+or if the environment hasn't changed, in the child, do
+
+    POSIX::setlocale(LC_ALL, "");
+
 =item Environment variables
 
 Currently, on all platforms except MSWin32, all I<system> calls (e.g., using

dist/threads/threads.xs

@@ -580,6 +580,8 @@ S_ithread_run(void * arg)
     S_set_sigmask(&thread->initial_sigmask);
 #endif
 
+    thread_locale_init();
+
     PL_perl_destruct_level = 2;
 
     {
@@ -665,6 +667,8 @@ S_ithread_run(void * arg)
     MUTEX_UNLOCK(&thread->mutex);
     MUTEX_UNLOCK(&MY_POOL.create_destruct_mutex);
 
+    thread_locale_term();
+
     /* Exit application if required */
     if (exit_app) {
         (void)S_jmpenv_run(aTHX_ 2, thread, NULL, &exit_app, &exit_code);

embed.fnc

@@ -1310,6 +1310,8 @@ Xp	|void	|set_numeric_underlying
 Xp	|void	|set_numeric_standard
 Xp	|bool	|_is_in_locale_category|const bool compiling|const int category
 Apd	|void	|sync_locale
+ApMn	|void	|thread_locale_init
+ApMn	|void	|thread_locale_term
 ApdO	|void	|require_pv	|NN const char* pv
 Apd	|void	|pack_cat	|NN SV *cat|NN const char *pat|NN const char *patend \
 				|NN SV **beglist|NN SV **endlist|NN SV ***next_in_list|U32 flags
@@ -2796,6 +2798,12 @@ s	|void	|new_collate	|NULLOK const char* newcoll
 s	|void	|new_ctype	|NN const char* newctype
 s	|void	|set_numeric_radix|const bool use_locale
 s	|void	|new_numeric	|NULLOK const char* newnum
+#    ifdef USE_POSIX_2008_LOCALE
+sn	|const char*|emulate_setlocale|const int category		\
+				    |NULLOK const char* locale		\
+				    |unsigned int index			\
+				    |const bool is_index_valid
+#    endif
 #    ifdef WIN32
 s	|char*	|win32_setlocale|int category|NULLOK const char* locale
 #    endif

embed.h

@@ -717,6 +717,8 @@
 #define sync_locale()		Perl_sync_locale(aTHX)
 #define taint_env()		Perl_taint_env(aTHX)
 #define taint_proper(a,b)	Perl_taint_proper(aTHX_ a,b)
+#define thread_locale_init	Perl_thread_locale_init
+#define thread_locale_term	Perl_thread_locale_term
 #define to_uni_lower(a,b,c)	Perl_to_uni_lower(aTHX_ a,b,c)
 #define to_uni_lower_lc(a)	Perl_to_uni_lower_lc(aTHX_ a)
 #define to_uni_title(a,b,c)	Perl_to_uni_title(aTHX_ a,b,c)
@@ -1635,6 +1637,9 @@
 #define new_numeric(a)		S_new_numeric(aTHX_ a)
 #define set_numeric_radix(a)	S_set_numeric_radix(aTHX_ a)
 #define stdize_locale(a)	S_stdize_locale(aTHX_ a)
+#      if defined(USE_POSIX_2008_LOCALE)
+#define emulate_setlocale	S_emulate_setlocale
+#      endif
 #      if defined(WIN32)
 #define win32_setlocale(a,b)	S_win32_setlocale(aTHX_ a,b)
 #      endif

embedvar.h

@@ -106,6 +106,7 @@
 #define PL_cryptseen		(vTHX->Icryptseen)
 #define PL_curcop		(vTHX->Icurcop)
 #define PL_curcopdb		(vTHX->Icurcopdb)
+#define PL_curlocales		(vTHX->Icurlocales)
 #define PL_curpad		(vTHX->Icurpad)
 #define PL_curpm		(vTHX->Icurpm)
 #define PL_curpm_under		(vTHX->Icurpm_under)

ext/POSIX/lib/POSIX.pod

@@ -939,6 +939,7 @@ containing the current underlying locale's formatting values.  Users of this fun
 should also read L<perllocale>, which provides a comprehensive
 discussion of Perl locale handling, including
 L<a section devoted to this function|perllocale/The localeconv function>.
+Prior to Perl 5.28, or when operating in a non thread-safe environment,
 It should not be used in a threaded application unless it's certain that
 the underlying locale is C or POSIX.  This is because it otherwise
 changes the locale, which globally affects all threads simultaneously.

intrpvar.h

@@ -576,7 +576,15 @@ PERLVAR(I, constpadix,	PADOFFSET)	/* lowest unused for constants */
 
 PERLVAR(I, padix_floor,	PADOFFSET)	/* how low may inner block reset padix */
 
+#if defined(USE_POSIX_2008_LOCALE)          \
+ && defined(USE_THREAD_SAFE_LOCALE)         \
+ && ! defined(HAS_QUERYLOCALE)
+
+PERLVARA(I, curlocales, 12, char *)
+
+#endif
 #ifdef USE_LOCALE_COLLATE
+
 PERLVAR(I, collation_name, char *)	/* Name of current collation */
 PERLVAR(I, collxfrm_base, Size_t)	/* Basic overhead in *xfrm() */
 PERLVARI(I, collxfrm_mult,Size_t, 2)	/* Expansion factor in *xfrm() */