3 case studies for -Zmiri-tag-raw-pointers

[dtolnay]

Following @saethlin's and @5225225's PRs dtolnay/ryu#37 and dtolnay/ryu#38, I've been trying -Zmiri-tag-raw-pointers on some of my repos containing unsafe code. So far the findings mostly line up with my expectations — I just had to fix a few easy cases similar to the PR above (array.get_unchecked(i) ⟶ array.as_ptr().offset(i)).

I found 3 cases of more complicated data structures that I would love to get working with -Zmiri-tag-raw-pointers. I am opening this issue not primarily because I want someone to come fix my broken code, but in case these are helpful real-world data structures to inform the implementation of miri-tag-raw-pointers or the resolution of rust-lang/unsafe-code-guidelines#148 and rust-lang/unsafe-code-guidelines#194, which looked the most relevant to me. Please go ahead and close if you are chock full of controvertible code already and these are not going to be useful.

---

https://github.com/dtolnay/syn TokenBuffer

MIRIFLAGS=-Zmiri-tag-raw-pointers cargo miri test --all-features --test test_item

error: Undefined Behavior: no item granting read access to tag <408023> at alloc154479+0x40 found in borrow stack.
   --> /git/syn/src/buffer.rs:191:19
    |
191 |         while let Entry::End(exit) = *ptr {
    |                   ^^^^^^^^^^^^^^^^ no item granting read access to tag <408023> at alloc154479+0x40 found in borrow stack.
    |
    = help: this indicates a potential bug in the program: it performed an invalid operation, but the rules it violated are still experimental
    = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
            
    = note: inside `syn::buffer::Cursor::create` at /git/syn/src/buffer.rs:191:19
    = note: inside `syn::buffer::Cursor::bump` at /git/syn/src/buffer.rs:214:9
    = note: inside `syn::buffer::Cursor::group` at /git/syn/src/buffer.rs:256:66
    = note: inside closure at /git/syn/src/group.rs:75:46
    = note: inside `syn::parse::ParseBuffer::step::<[closure@syn::group::parse_delimited::{closure#0}], (proc_macro2::Span, syn::parse::ParseBuffer)>` at /git/syn/src/parse.rs:1024:28
    = note: inside `syn::group::parse_delimited` at /git/syn/src/group.rs:74:5
    = note: inside `syn::group::parse_brackets` at /git/syn/src/group.rs:56:5
    = note: inside `syn::attr::parsing::single_parse_outer` at /git/syn/src/group.rs:272:15
    = note: inside `syn::parse::ParseBuffer::call::<syn::Attribute>` at /git/syn/src/parse.rs:505:9
    = note: inside `syn::Attribute::parse_outer` at /git/syn/src/attr.rs:241:24
    = note: inside `syn::parse::ParseBuffer::call::<std::vec::Vec<syn::Attribute>>` at /git/syn/src/parse.rs:505:9
    = note: inside `syn::item::parsing::<impl syn::parse::Parse for syn::Item>::parse` at /git/syn/src/item.rs:994:29
    = note: inside `<for<'r> fn(&'r syn::parse::ParseBuffer<'r>) -> std::result::Result<syn::Item, syn::Error> {<syn::Item as syn::parse::Parse>::parse} as std::ops::FnOnce<(&syn::parse::ParseBuffer,)>>::call_once - shim(for<'r> fn(&'r syn::parse::ParseBuffer<'r>) -> std::result::Result<syn::Item, syn::Error> {<syn::Item as syn::parse::Parse>::parse})` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `<for<'r> fn(&'r syn::parse::ParseBuffer<'r>) -> std::result::Result<syn::Item, syn::Error> {<syn::Item as syn::parse::Parse>::parse} as syn::parse::Parser>::parse2` at /git/syn/src/parse.rs:1209:20
    = note: inside `syn::parse2::<syn::Item>` at /git/syn/src/lib.rs:913:5
note: inside `<proc_macro2::TokenStream as macros::Tokens>::parse::<syn::Item>` at tests/macros/mod.rs:79:9
   --> tests/macros/mod.rs:79:9
    |
79  |         syn::parse2(self)
    |         ^^^^^^^^^^^^^^^^^
note: inside `test_macro_variable_attr` at tests/macros/mod.rs:40:21
   --> tests/macros/mod.rs:40:21
    |
40  |           let $expr = crate::macros::Tokens::parse::<$t>($expr).unwrap();
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
   ::: tests/test_item.rs:20:5
    |
20  | /     snapshot!(tokens as Item, @r###"
21  | |     Item::Fn {
22  | |         attrs: [
23  | |             Attribute {
...   |
43  | |     }
44  | |     "###);
    | |_________- in this macro invocation
note: inside closure at tests/test_item.rs:10:1
   --> tests/test_item.rs:10:1
    |
9   |   #[test]
    |   ------- in this procedural macro expansion
10  | / fn test_macro_variable_attr() {
11  | |     // mimics the token stream corresponding to `$attr fn f() {}`
12  | |     let tokens = TokenStream::from_iter(vec![
13  | |         TokenTree::Group(Group::new(Delimiter::None, quote! { #[test] })),
...   |
44  | |     "###);
45  | | }
    | |_^
    = note: inside `<[closure@tests/test_item.rs:10:1: 45:2] as std::ops::FnOnce<()>>::call_once - shim` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `test::__rust_begin_short_backtrace::<fn()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:585:5
    = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:576:30
    = note: inside `<[closure@test::run_test::{closure#2}] as std::ops::FnOnce<()>>::call_once - shim(vtable)` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send> as std::ops::FnOnce<()>>::call_once` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1811:9
    = note: inside `<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>> as std::ops::FnOnce<()>>::call_once` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9
    = note: inside `std::panicking::r#try::do_call::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
    = note: inside `std::panicking::r#try::<(), std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
    = note: inside `std::panic::catch_unwind::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
    = note: inside `test::run_test_in_process` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:608:18
    = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:500:39
    = note: inside `test::run_test::run_test_inner` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:538:13
    = note: inside `test::run_test` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:572:28
    = note: inside `test::run_tests::<[closure@test::run_tests_console::{closure#2}]>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:313:17
    = note: inside `test::run_tests_console` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/console.rs:290:5
    = note: inside `test::test_main` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:124:15
    = note: inside `test::test_main_static` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:143:5
    = note: inside `main`
    = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `std::sys_common::backtrace::__rust_begin_short_backtrace::<fn(), ()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:123:18
    = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:145:18
    = note: inside `std::ops::function::impls::<impl std::ops::FnOnce<()> for &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>::call_once` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:259:13
    = note: inside `std::panicking::r#try::do_call::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
    = note: inside `std::panicking::r#try::<i32, &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
    = note: inside `std::panic::catch_unwind::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
    = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128:48
    = note: inside `std::panicking::r#try::do_call::<[closure@std::rt::lang_start_internal::{closure#2}], isize>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
    = note: inside `std::panicking::r#try::<isize, [closure@std::rt::lang_start_internal::{closure#2}]>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
    = note: inside `std::panic::catch_unwind::<[closure@std::rt::lang_start_internal::{closure#2}], isize>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
    = note: inside `std::rt::lang_start_internal` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128:20
    = note: inside `std::rt::lang_start::<()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:144:17
    = note: this error originates in the macro `snapshot_impl` (in Nightly builds, run with -Z macro-backtrace for more info)

---

https://github.com/dtolnay/semver Identifier

MIRIFLAGS=-Zmiri-tag-raw-pointers cargo miri test --test test_identifier

error: Undefined Behavior: no item granting read access to tag <untagged> at alloc64891 found in borrow stack.
   --> /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:701:9
    |
701 |         copy_nonoverlapping(src, tmp.as_mut_ptr(), 1);
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no item granting read access to tag <untagged> at alloc64891 found in borrow stack.
    |
    = help: this indicates a potential bug in the program: it performed an invalid operation, but the rules it violated are still experimental
    = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
            
    = note: inside `std::ptr::read::<[u8; 2]>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:701:9
    = note: inside `semver::identifier::decode_len` at /git/semver/src/identifier.rs:301:36
    = note: inside `<semver::identifier::Identifier as std::ops::Drop>::drop` at /git/semver/src/identifier.rs:200:28
    = note: inside `std::ptr::drop_in_place::<semver::identifier::Identifier> - shim(Some(semver::identifier::Identifier))` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:188:1
    = note: inside `std::ptr::drop_in_place::<semver::Prerelease> - shim(Some(semver::Prerelease))` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:188:1
note: inside `test_eq` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/macros/mod.rs:97:5
   --> tests/test_identifier.rs:41:5
    |
41  |     assert_ne!(prerelease("aaaaaaaaa"), prerelease("a"));
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure at tests/test_identifier.rs:37:1
   --> tests/test_identifier.rs:37:1
    |
36  |   #[test]
    |   ------- in this procedural macro expansion
37  | / fn test_eq() {
38  | |     assert_eq!(prerelease("-"), prerelease("-"));
39  | |     assert_ne!(prerelease("a"), prerelease("aa"));
40  | |     assert_ne!(prerelease("aa"), prerelease("a"));
...   |
44  | |     assert_ne!(build_metadata("1"), build_metadata("001"));
45  | | }
    | |_^
    = note: inside `<[closure@tests/test_identifier.rs:37:1: 45:2] as std::ops::FnOnce<()>>::call_once - shim` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `test::__rust_begin_short_backtrace::<fn()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:585:5
    = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:576:30
    = note: inside `<[closure@test::run_test::{closure#2}] as std::ops::FnOnce<()>>::call_once - shim(vtable)` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send> as std::ops::FnOnce<()>>::call_once` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1811:9
    = note: inside `<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>> as std::ops::FnOnce<()>>::call_once` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9
    = note: inside `std::panicking::r#try::do_call::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
    = note: inside `std::panicking::r#try::<(), std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
    = note: inside `std::panic::catch_unwind::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
    = note: inside `test::run_test_in_process` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:608:18
    = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:500:39
    = note: inside `test::run_test::run_test_inner` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:538:13
    = note: inside `test::run_test` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:572:28
    = note: inside `test::run_tests::<[closure@test::run_tests_console::{closure#2}]>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:313:17
    = note: inside `test::run_tests_console` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/console.rs:290:5
    = note: inside `test::test_main` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:124:15
    = note: inside `test::test_main_static` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:143:5
    = note: inside `main`
    = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `std::sys_common::backtrace::__rust_begin_short_backtrace::<fn(), ()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:123:18
    = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:145:18
    = note: inside `std::ops::function::impls::<impl std::ops::FnOnce<()> for &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>::call_once` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:259:13
    = note: inside `std::panicking::r#try::do_call::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
    = note: inside `std::panicking::r#try::<i32, &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
    = note: inside `std::panic::catch_unwind::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
    = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128:48
    = note: inside `std::panicking::r#try::do_call::<[closure@std::rt::lang_start_internal::{closure#2}], isize>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
    = note: inside `std::panicking::r#try::<isize, [closure@std::rt::lang_start_internal::{closure#2}]>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
    = note: inside `std::panic::catch_unwind::<[closure@std::rt::lang_start_internal::{closure#2}], isize>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
    = note: inside `std::rt::lang_start_internal` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128:20
    = note: inside `std::rt::lang_start::<()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:144:17
    = note: this error originates in the macro `assert_ne` (in Nightly builds, run with -Z macro-backtrace for more info)

---

https://github.com/dtolnay/miniserde Deserializer

cargo miri test --test test_derive

error: Undefined Behavior: trying to reborrow for SharedReadWrite at alloc64094, but parent tag <165391> does not have an appropriate item in the borrow stack
  --> /git/miniserde/src/json/de.rs:88:17
   |
88 |                 visitor.string(s)?;
   |                 ^^^^^^^^^^^^^^^^^ trying to reborrow for SharedReadWrite at alloc64094, but parent tag <165391> does not have an appropriate item in the borrow stack
   |
   = help: this indicates a potential bug in the program: it performed an invalid operation, but the rules it violated are still experimental
   = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
           
   = note: inside `miniserde::json::de::from_str_impl` at /git/miniserde/src/json/de.rs:88:17
   = note: inside `miniserde::json::from_str::<Example>` at /git/miniserde/src/json/de.rs:32:5
note: inside `test_de` at tests/test_derive.rs:27:27
  --> tests/test_derive.rs:27:27
   |
27 |     let actual: Example = json::from_str(j).unwrap();
   |                           ^^^^^^^^^^^^^^^^^
note: inside closure at tests/test_derive.rs:25:1
  --> tests/test_derive.rs:25:1
   |
24 |   #[test]
   |   ------- in this procedural macro expansion
25 | / fn test_de() {
26 | |     let j = r#" {"x": "X", "t1": "A", "t2": "renamedB", "n": {"y": ["Y", "Y"]}} "#;
27 | |     let actual: Example = json::from_str(j).unwrap();
28 | |     let expected = Example {
...  |
37 | |     assert_eq!(actual, expected);
38 | | }
   | |_^
   = note: inside `<[closure@tests/test_derive.rs:25:1: 38:2] as std::ops::FnOnce<()>>::call_once - shim` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
   = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
   = note: inside `test::__rust_begin_short_backtrace::<fn()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:585:5
   = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:576:30
   = note: inside `<[closure@test::run_test::{closure#2}] as std::ops::FnOnce<()>>::call_once - shim(vtable)` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
   = note: inside `<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send> as std::ops::FnOnce<()>>::call_once` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1811:9
   = note: inside `<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>> as std::ops::FnOnce<()>>::call_once` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9
   = note: inside `std::panicking::r#try::do_call::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
   = note: inside `std::panicking::r#try::<(), std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
   = note: inside `std::panic::catch_unwind::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
   = note: inside `test::run_test_in_process` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:608:18
   = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:500:39
   = note: inside `test::run_test::run_test_inner` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:538:13
   = note: inside `test::run_test` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:572:28
   = note: inside `test::run_tests::<[closure@test::run_tests_console::{closure#2}]>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:313:17
   = note: inside `test::run_tests_console` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/console.rs:290:5
   = note: inside `test::test_main` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:124:15
   = note: inside `test::test_main_static` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:143:5
   = note: inside `main`
   = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
   = note: inside `std::sys_common::backtrace::__rust_begin_short_backtrace::<fn(), ()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:123:18
   = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:145:18
   = note: inside `std::ops::function::impls::<impl std::ops::FnOnce<()> for &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>::call_once` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:259:13
   = note: inside `std::panicking::r#try::do_call::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
   = note: inside `std::panicking::r#try::<i32, &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
   = note: inside `std::panic::catch_unwind::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
   = note: inside closure at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128:48
   = note: inside `std::panicking::r#try::do_call::<[closure@std::rt::lang_start_internal::{closure#2}], isize>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
   = note: inside `std::panicking::r#try::<isize, [closure@std::rt::lang_start_internal::{closure#2}]>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
   = note: inside `std::panic::catch_unwind::<[closure@std::rt::lang_start_internal::{closure#2}], isize>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
   = note: inside `std::rt::lang_start_internal` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128:20
   = note: inside `std::rt::lang_start::<()>` at /home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:144:17
   = note: this error originates in the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

---

[RalfJung]

Thanks for sharing!

So do all of these involve self-referential structs or 'owning-ref'-style pointers? (I am asking since you mentioned rust-lang/unsafe-code-guidelines#148 and rust-lang/unsafe-code-guidelines#194.)

[saethlin]

I don't have any comment on the syn example. I'm having some trouble tracing the code in the few minutes I spent looking through each of these examples.

---

In the semver example, I think the responsible code is https://github.com/dtolnay/semver/blob/9a2084502fb595a1e016a9c72d333956664e0c48/src/identifier.rs#L241-L247

// Shift out the 1 previously placed into the most significant bit of the least
// significant byte. Shift in a low 0 bit to reconstruct the original 2-byte
// aligned pointer.
fn repr_to_ptr(repr: NonZeroU64) -> *const u8 {
    // `lea rax, [rdi + rdi]`
    (repr.get() << 1) as *const u8
}

This looks like an int-to-pointer cast. Or perhaps it even counts as an OOTA pointer? rust-lang/unsafe-code-guidelines#291 doesn't mention doing arithmetic on an integer before converting it back to a pointer.

---

I think the miniserde one is totally beyond me. I suspect that what's at play here is a self-referential struct, which is made to compile with this code:
https://github.com/dtolnay/miniserde/blob/17c036fdcf54d00fb1f06a53a1b8db75694205db/src/place.rs#L25-L39

#[macro_export]
macro_rules! make_place {
    ($name:ident) => {
        #[repr(C)]
        struct $name<__T> {
            out: $crate::__private::Option<__T>,
        }

        impl<__T> $name<__T> {
            fn new(out: &mut $crate::__private::Option<__T>) -> &mut Self {
                unsafe { &mut *{ out as *mut $crate::__private::Option<__T> as *mut $name<__T> } }
            }
        }
    };
}

But in any case, this miniserde problem is a great case study for how SB needs better diagnostics. What I'm confident in is that Miri detects the error far from the invalid-by-SB unsafe code.

[RalfJung]

This looks like an int-to-pointer cast.

Ah yes. That is very hard to support with tag-raw-pointers. This seems similar to #1866 (comment) (ferrilab/bitvec#135).

Or perhaps it even counts as an OOTA pointer? rust-lang/unsafe-code-guidelines#291 doesn't mention doing arithmetic on an integer before converting it back to a pointer.

Presumably the arithmetic here results in exactly the same integer value as what was returned by a prior ptr-to-int cast. The problem is more that casting an int to a ptr is inherently a rather cursed operation (for memory that is 'known' to the language -- so this statement excludes accessing non-Rust-managed memory at fixed addresses; that is fine as long as the aliasing rules are otherwise followed). The cast can often be avoided by using something like this, but it can require significant refactoring:

// Converts 'addr' to a pointer using the provenance of 'prov'.
fn int_to_ptr_with_provenance<T>(addr: usize, prov: *const T) -> *const T {
  // Nowhere in this function do we cast an integer to a pointer!
  let ptr = prov.cast::<u8>();
  ptr.wrapping_add(addr.wrapping_sub(ptr as usize)).cast()
}

In this case it looks like NonZeroU64 is used to hold data that might have provenance... I would usually recommend NonNull<()> instead but that would have a different size on 32bit systems so I doubt it will work here. Or alternatively a union of a pointer and an integer, but that would kill the niche. Not sure if we have a good option with current Rust here.

[dtolnay]

So do all of these involve self-referential structs or 'owning-ref'-style pointers?

miniserde and semver probably count as that. syn is maybe? The data structure is an acyclic singly linked list of T across multiple Box<[T]> allocations.

[camelid]

(What does OOTA stand for?)

[saethlin]

(What does OOTA stand for?)

Out Of Thin Air: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1217r0.html

[saethlin]

Just a minor update: It looks like in the meantime @dtolnay patched miniserde so that it is compatible with raw pointer tagging (though where exactly I'm not quite sure), and I've put up PRs to semver and syn which whether or not they meet maintainer approval, pass Miri without changing the underlying data structures or degrading performance.

So I guess whether or not you say

I am opening this issue not primarily because I want someone to come fix my broken code

I do indeed struggle to resist this kind of challenge over the long term :p

[saethlin]

PRs to semver: dtolnay/semver#273 and syn: dtolnay/syn#1166 are now merged.

I think we could close this now?

[RalfJung]

Looks like it. Thanks a ton @saethlin :)