Provide examples

[SuperFluffy]

What's the current state of this? It seems like out of all the hdf5 bindings out there, yours seems to be the most actively developed.

Are you planning to provide some example on how to interact with hdf5-rs, or do you think it is not ready for consumption yet?

[aldanor]

I'm planning to work on the datasets reading/writing in February, at which point it will become fully useable. I've put quite a lot of effort to make the bindings thread-safe (which other wrapping libraries don't really care about), this includes locking operations with reentrant mutexes and providing some helper macros.

I guess for now you can look at the tests for each module (file/group) to see how it works.

There's one major stumbling point (or a problem of choice, rather) which prevented me from implementing datasets earlier -- how exactly to deal with reading/writing structured datasets, I don't have a good idea of how the API should look like (given there's no proper runtime struct introspection in Rust).

[aldanor]

I was also thinking of using rust-ndarray as the in-memory backend of choice, now that it's actively developed: https://github.com/bluss/rust-ndarray

[mokasin]

@aldanor Maybe I don't understand correctly, what you are getting at. But why do you need runtime introspection? Wouldn't rust-serialize or serde help you, to do the job at compile time?

[aldanor]

@mokasin Because that's how HDF5 works:

• datasets are homogeneous collections of data
• you have to create datasets explicitly before pushing any data to them, which requires specifying the memory layout
• you're unlikely to serialize data item-by-item, the whole point of using HDF5 is to able to read/write large amounts of similarly-typed objects at once
• you can kind of hack around serde/rustc-serialize to get field names etc for compound types, but you won't get field offsets without further hacks; all in all it's not worth it

As a matter of fact, I've written a little utility package which should help with all this: https://github.com/aldanor/typeinfo

[Enet4]

I was particularly interested in reading datasets from HDF5 files, for which so far there is no high-level API for Rust. Is there any progress on this particular feature?

[aldanor]

@Enet4 As a matter of fact, I'm sort of working on that right now because I need it too :)

There's a few stumbling blocks to which I have no particular solutions yet, for instance how to represent the variable-length types (strings in particular) -- and whether to support them at all.

[aldanor]

It's more of a matter of figuring things out than anything. If anyone in this thread has time/desire to discuss possible ways of handling compound / vlen datatypes in Rust -- this could speed things up 😉

[Enet4]

I'm a beginner on Rust, unfortunately. Still, I am interested in this and would like to keep in touch with any discussion that will hopefully take place, perhaps even managing to contribute with my thoughts.

[aldanor]

The main question is what to do with types that are not Copy or with variable-length datatypes in HDF5 if you want to get some objects with expected Rust semantics in the end. E.g, a variable-length string as a field in a compound datatype.

For element of a variable-length datatype, I think H5Dread will allocate structs like this

struct hvl_t {
    len: size_t,
    p: *mut c_void,
}

However it then has to be somehow translated to Rust structs with proper drop semantics etc.

[aldanor]

Wrapping structs like this in Rust is possible, but if there's a Drop impl, Rust compiler will add an extra field so the struct cannot have the same layout is in C (#[repr(C)]). This will make it incompatible with what HDF5 expects.

As of today, the only way to avoid that is to add #![feature(unsafe_no_drop_flag)] crate attribute which will enable using #[unsafe_no_drop_flag]. This will imply that hdf5-rs would no longer be compatible with stable Rust -- for some time, until the Rust compiler team implements "dynamic drop" semantics which they are planning to do in the foreseeable future (issue 5016 on rust-lang/rust).

I wonder if that'd be an acceptable solution.

[aldanor]

Proof of concept: https://gist.github.com/aldanor/8ca3bb807c79836455a87c699d844efe

[mokasin]

IMHO: If this is the only feasible solution, I'd say go for it. A crate only working on nightly is better than a non working crate. And either this stabilise itself or someone finds a different solution later.

[Enet4]

Wrapping structs like this in Rust is possible, but if there's a Drop impl, Rust compiler will add an extra field so the struct cannot have the same layout is in C (#[repr(C)]). This will make it incompatible with what HDF5 expects.

Couldn't there be an implicit conversion between a type with normal Rust semantics and an internal, "unsafe" implementation whenever that is needed? Or am I just missing the point?

[aldanor]

@Enet4 Would you care to elaborate on what exactly do you mean by "implicit conversion"? Could you provide an example?

Imagine a case where you read the dataset where the datatype is compound, one field is an int, another a variable-length string. H5Dread will write into a buffer that looks like this:

struct A {
    int x;     // normal field
    struct {   // vlen field
        void *p;
        size_t len;
    } y;
};

Now assume you have 1 billion of these (so calling 1 billion constructors is out of question) taking almost all of your RAM (so copying all of the data is out of question), you have a C pointer to this data which you received from HDF5, what would you do next to construct a zero-copy Rust view on this which behaves like a bunch of normal Rust objects and ensures the proper cleanup?

For each of the variable-length elements, HDF5 will malloc() the memory and leave it to the user to free. This implies that the wrapper type for y field in Rust will need to be droppable and should call free() when dropped. However, Drop is not compatible with #[repr(C)] as it essentially adds a third 8-byte element to the end of y and stores the drop flag there indicating destructor status. Therefore, the struct doesn't have the same layout as C anymore. You can hack around this by adjusting the field offsets etc, but what's worse, you cannot safely memset this struct or pass it to the C library as is since the hidden drop byte may be affected.

The only way to have both Drop and #[repr(C)] avoiding the drop flags is the #[unsafe_no_drop_flag] as of today which requires nightly channel.

The user-code would look like this, I think:

h5def! {                  // macro that enables extracting offset/type information at runtime
    pub struct A {
        pub x: u32,
        pub y: VLString,  // variable-length zero-copy string view on a malloc'd buffer
    }
}

[clamydo]

Does this also mean, that a user of this crate would have to drop the data himself?