Experimental support for unicode identifiers #1499
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rebase of #1407. Copied from there:
I know for a fact that this requires a few changes in
stan-dev/stan
's json data handler to recognize unicode names, which is just one of several reasons this is a draft.The basic overview:
OCaml strings should be treated mostly like arrays of bytes, and
ocamllex
handles inputs as sets of bytes. We can define rules that recognize UTF-8-compatible bytes, and then do validation on them after the fact based on the the Unicode Annex 31: Unicode Identifiers standard.We then pretend for most of the compiler like it's just bytes, which is fine, because we never do things like subslice variable names.
Finally, at output time, we already had string escaping (since #952), so most of the code-gen works fine. Recent C++ standards require that compilers support UTF-8 names based on the same UAX31 rules linked above, but older ones may not. For now I've got it generating "Universal character names" which seem like the legacy version of this, which hopefully means older compilers will be happy with it.
Submission Checklist
Release notes
stanc3 can now accept a flag --allow-unicode which enables the use of non-ascii characters in Stan files. All files are expected to be encoded in UTF-8.
This is experimental and may not work with older C++ compilers.
Copyright and Licensing
By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the BSD 3-clause license (https://opensource.org/licenses/BSD-3-Clause)