Reading / writing

This may be a stupid question, but I see the Dataset.read method being used in your example code in issues like #9 and I can't actually find documentation, or source code, for it. :)

I guess examples of use would be very helpful, as in #9, but I thought at least I could find the code to help me understand things! I have lots of questions, about what kinds of structured-types can be stored in datasets, whether datasets can be iterated to avoid loading them all into memory, etc. etc.; happy to answer these questions myself if I know where to look.

Thanks for what appears to be a very complete and powerful library!

[Enet4]

I was once interested in this library for exactly that purpose. It happens that the code for reading from an existing dataset is in a separate branch, and is currently not documented. It might also be incomplete for certain data types.

Maybe the owner could use some help. We might be able to join forces around this functionality and document the working parts for future users.

[aldanor]

@Enet4 @cathalgarvey Hi guys!

@Enet4 is correct, the type mapping is in a separate branch (types). I've rewritten it at least 3 times and it's pretty much complete -- supports all builtin types, tuple types, fixed-size array types, all 4 kinds of strings (fixed/varlen, ascii/utf-8 -- this already makes it support more atomics than e.g. pytables / h5py), plus it uses procedural macros to bind structs and nested structs, like so #[derive(H5Type)]. Last time I've had free time was this spring -- which is when I've managed to complete the type system. The next time will be this fall / end of summer -- could try to implement the next batch of features to make this usable.

This way, most of the super-low level stuff that deals with types is already done. The next one is Dataset API. I have a few implementations but I don't like neither of them too much for different reasons. One is Dataset<T: H5Type> where the element type is a type parameter. However, this way you lose the ability to inspect dataset header without caring for its type (e.g. check size / shape / ndim / filters and other stuff not depending on the type). The other way is having two types for datasets, and a common trait, this way things get pretty complicated type-wise very fast. Another way is having MaybeH5Type trait and using Dataset<T: MaybeH5Type>, which is also quite ugly, etc...

The ideal thing would be to get rid of all traits altogether (e.g. Container, Location) and use specializations instead; so you have things like type Dataset = Object<DatasetID>, type UntypedDataset = Object<UntypedDatasetID>. I've started doing work on that but then, after it was done, stumbled upon this unfortunate rustdoc bug rust-lang/rust#32077 which made all docs disappear. Very sad since that would definitely be the best solution. I don't have sufficient rust internals knowledge to fix it upstream, the best thing I can think of is opening a bounty on bountysource :)

Yes I could def. use some help -- I've no problem implementing things, and I know HDF5 core C API very well, most of the pain is in decision making.

[Enet4]

Hmm, so let's think...

I have a few implementations but I don't like neither of them too much for different reasons. One is Dataset<T: H5Type> where the element type is a type parameter. However, this way you lose the ability to inspect dataset header without caring for its type (e.g. check size / shape / ndim / filters and other stuff not depending on the type).

I would imagine having an implementation of H5Type on a new type with dynamic type fetching (DynValueType or something). Is there something discarding this possibility right now?

Another way is having MaybeH5Type trait and using Dataset<T: MaybeH5Type>, etc...

But yes, I suppose if the above can't be done, a more relaxed trait could be used.

The ideal thing would be to get rid of all traits altogether (e.g. Container, Location) and use specializations instead; so you have things like type Dataset = Object, type UntypedDataset = Object. I've started doing work on that but then stumbled upon this unfortunate rustdoc bug rust-lang/rust#32077 which made all docs disappear. Very sad since that would definitely be the best solution.

I don't know much about how the Object API currently works, so I'll just ask for some clarifications here. Are you suggesting some kind of interface inheritance based on the parameter type? Is this preferable over using supertraits (e.g. trait Dataset: Object { ... })?

[aldanor]

I would imagine having an implementation of H5Type on a new type with dynamic type fetching (DynValueType or something). Is there something discarding this possibility right now?

I'm not sure I follow, would you care to expand?

I don't know much about how the Object API currently works, so I'll just ask for some clarifications here. Are you suggesting some kind of interface inheritance based on the parameter type? Is this preferable over using supertraits (e.g. trait Dataset: Object { ... })?

Basically yes, interface inheritance based on the parameter type. Currently, there are a bunch of traits like Object, Location, Container, etc (loosely following class hierarchy of HDF5 C++ API), sort of like this:

trait ID { fn id() ... }
trait FromID {}

// note that these traits are "dummy", they don't have any methods to implement
trait Object : ID {}
trait Location : Object {}
trait Container : Location {}

struct Group { id: ... }
impl ID for Group { fn id(): ... }
impl Object for Group {}
impl Location for Group {}
impl Container for Group {}

struct PropertyList { id: ... }
impl ID for PropertyList { fn id(): ... }
impl Object for PropertyList {}

There are a few problems with a trait-based inheritance here:

1. All HDF5 objects have the same data in them -- it's just the integer ID of the object. But because there's no data inheritance in Rust, each and every concrete object type (e.g. Group) has to store the ID and implement some trait (ID) that extracts it and another trait (FromID) that tries to create the object from an existing ID. It's quite a lot of copy/paste and duplication, definitely more than I'd like -- for an example, see e.g. this.

2. Downstream code has to bring all these traits into user namespace which is far from ideal and makes the API hard to explore (this is unlike the C++ or Python API). For instance,

   let group = file.group("foo")?;
   println!("{}", group.name());  // ~ERROR: must bring hdf5_rs::Location into scope

3. Requires creating "intermediate" traits -- for example, in the case of dataset, there must be concrete types for typed / untyped datasets --> hence there must be a AnyDataset trait (for the lack of better name), because that's the only way we can share the functionality between interfaces.

There're definitely more problems but these are the ones that come to mind first. This just feels like misusing Rust's traits (given that most of them have empty interfaces anyway, they're just "dummy").

What I wanted to do is inheritance based on type parameter so some of the problems listed above go away, like shown here: rust-lang/rust#32077 (comment). And it kind of worked if it wasn't for the rustdoc woes.

[aldanor]

// I've opened a bounty on bountysource for the rustdoc issue, who knows, maybe someone will get to fixing it :)

[thibaultbarbie]

I am also interested to an example of how to read a dataset. My hdf5 dataset is really simple, just some groups with matrix inside. I am still a rust beginner and could not find how to do it just by reading the documentation. Could you please provide a minimal example ?

[Enet4]

As described here and in a few other issues (#9, #17, #18), dataset reading/writing is yet to be exposed at the high-level abstraction. I noticed that @aldanor has been working on the crate recently, so I hope to see this become available soon-ish. 🤞

[aldanor]

It's funny to find out someone's still tracking this :) Yep, after a long pause I've resumed work on the crate (I need it too and I need it working soon, myself!), you can see the current work in https://github.com/aldanor/hdf5-rs/tree/feature/2018. The crate now fully supports HDF5 1.10.0 and 1.10.1, too, on all platforms.

As a matter of fact, I already have a local branch working that can read N-dimensional datasets into both ndarray and Vec, I'm now working on finalizing the API, adding the writing part etc. There's tons of work (e.g. type conversion logic etc), but I do expect to publish something working in the next few months or so.

The type system is already done, including procedural macros with #[derive] (examples in tests).

The object system discussed earlier in this thread is done too, via Deref (so that documentation would kind of work), all traits are gone now.

I understand everyone's eager to have something they can use, but so am I, and I'm trying to juggle development on this crate with personal life on a daily basis as you can observe from the commit history...

[pmarks]

Hi @aldanor. We (@10XGenomics) are quite interested in Rust HDF5 support. We'd be happy to test / contribute to high-level Read/Write implementations whenever you're ready to share your branch, even if it's in a raw form. The new branch looks great so far!

[aldanor]

Hi folks, here's a topic for some brainstorming and bikeshedding, apologies in advance for a huge wall of text :) If you are interested in speeding this up, any ideas are definitely welcome.

I'm currently trying to figure out what's the most ergonomic / least annoying while at the same point type-safe way of representing reads and writes. I have the code that does read n-dimensional arrays successfully but it needs to be wrapped nicely. For simplicity, let's say we're just talking about full-dataset/attribute reads and writes here.

Here's a few facts:

1. There's a trait (H5Type) for all Rust types that can be noop-converted both ways to HDF5 types. This includes all atomic types, all four types of HDF5 strings including variable-length, fixed array types, variable length aray types, plus user-defined compound types via #[derive(H5Type)].
2. There's a notion of a stored HDF5 type and a memory HDF5 type. Stored is what's in the HDF5 file; memory is what ends up in your memory. If stored and memory types are the same, there's no conversion (noop conversion).
3. Given two types (source/destination), there's 4 different options: there's no conversion path; no-op conversion is available (same types); hard conversion (compiler conversion); soft conversion (can be user-extended). For reads: source=stored, destination=memory; for writes: source=memory, destination=stored.
4. Unlike Python, where we can just dataset.read() and it just returns an array with a proper type (so a noop conversion will be used automatically because stored=memory), in Rust we have to specify a memory type manually.
5. HDF5 library will not throw any errors or whatever if a non-noop conversion path is used, it will just convert elements automatically. However the only zero-cost way of reading/writing is by using the noop conversion path (which requires memory=stored). It would be nice to keep it as a default and provide an easy way of throwing an error that warns of non-trivial conversion (unless the user explicitly requests it). An example of such conversion is if you read an int64 dataset into int8 memory storage and by default the HDF5 library will just silently clip all values between -128 and 127 -- I'd rather prefer it complain loudly. Actually, I bet many (most?) HDF5 users won't be able to tell you what exactly happens under the hood if you read an int64 dataset into an int8; or a struct with an int64 field into a similar struct with an int8 field.

How do we turn this into an API?

In the current temporary prototype, I have something like this (all Array symbols are from ndarray crate):

// both Attribute and Dataset dereference to Container
// Container dereferences into Location
impl Container {
    pub fn read_vec<T: H5Type>(&self) -> Result<Vec<T>> { ... }
    pub fn read_1d<T: H5Type>(&self) -> Result<Array1<T>> { ... }
    pub fn read_2d<T: H5Type>(&self) -> Result<Array2<T>> { ... }
    pub fn read_scalar<T: H5Type>(&self) -> Result<T> { ... }
    pub fn read_dyn<T: H5Type>(&self) -> Result<ArrayD<T>> { ... }
    pub fn read_arr<T: H5Type, D: Dimension>(&self) -> Result<Array<T, D>> { ... }
    // also all write_*() methods
}

When reading, it would check that the conversion path stored->memory is noop (that is, it would create an HDF5 datatype from T, then load the stored datatype, and figure out the conversion path between two).

There's a few drawbacks here:

• What if I want to use hard conversion? Soft conversion?
• Every time you read from the same dataset, you'll have to specify ::<T>() again and again, which may seem redundant.

... which leads to a possibility of having a TypedContainer<T> alongside the untyped container:

impl Container {
    pub fn read_vec<T: H5Type>(&self) -> Result<Vec<T>> { ... }
    pub fn of_type<T: H5Type>(&self) -> Result<TypedContainer<T>> { ... } // noop
    pub fn cast_soft<T: H5Type>(&self) -> Result<TypedContainer<T>> { ... }
    // etc
}

impl<T: H5Type> TypedContainer<T> {
    pub fn read_vec(&self) -> Result<Vec<T>> { ... }
    // etc
}

... but here's one problem: if TypedContainer handles reads and writes, a single conversion path is not sufficient anymore. Basically, because reads are storage->memory and writes are memory->storage, there do exist types for which conversion paths are not symmetric (e.g. soft one way and non-existent the other way). So, you would need something like cast_read_soft() and cast_write_soft() which already starts getting somewhat unwieldy (and btw there does exist an enum for conversion paths covering all three cases).

Then I was thinking, maybe it could be done builder-style? E.g. hypothetically, something like...

let _ = dataset.read::<T>()?.vec()?; // read() returns ReadContainer
let _ = dataset.read_soft::<T>()?.arr_2d()?;

As for writes, until trait specialization lands, we can't provide blanket impls for all H5Type-covered types, so for now it would have to be something like

let _ = dataset.write::<T>()?.scalar(&x)?; // write() returns WriteContainer
let _ = dataset.write_soft::<T>()?.arr(&arr)?;
let _ = dataset.write_hard::<T>()?.slice(&vec)?;

(It might be somewhat confusing that .read() doesn't actually read anything though, and .write() doesn't actually write anything)

For writes, there's ton of other options like compression etc, need to think of an ergonomic way to support that (which is currently supported only in the DatasetBuilder). At the very least, it should be possible to reshape the dataset on the fly -- e.g. if you provide the data in a slice/Vec but want to store it as a 2-D array, maybe something like

let _ = dataset.write::<T>()?.shape((10, 20)).slice(&vec)?;

Last thing to note, the methods above could coexist with their alternative forms, e.g. you could have all of these

let _ = group.read_vec::<T>("foo")?;
let _ = group.dataset("foo")?.read_vec::<T>()?;
let _ = group.dataset("foo")?.read::<T>()?.vec()?;

Something like that...

Anyways, there's tons of ways to approach this, but we need to pick one :)

[aldanor]

Actually, pinging @bluss too -- hope he doesn't mind :) (since we're planning to use ndarray as the core backend here anyway)

[pmarks]

Cool, thanks for the background!

As for writes, until trait specialization lands, we can't provide blanket impls for all H5Type-covered types, so for now it would have to be something like

let _ = dataset.write::<T>()?.scalar(&x)?; // write() returns WriteContainer
let _ = dataset.write_soft::<T>()?.arr(&arr)?;
let _ = dataset.write_hard::<T>()?.slice(&vec)?;

So to clarify, the proposal is that dataset.write::<T>() would return Ok(WriteContainer<T>) if there's a noop path from storage->T, and Err() otherwise. Whereas dataset.write_hard::<T>() would return Ok(WriteContainer<T>) if there was a noop or hard conversion available, but would return Err(<invalid conversion>) if only a soft conversion was available? I really like the typed reader / writer idea, and it doesn't seem to hurt ergonomics much to split TypedContainer into ReadContainer and WriterContainer. I'd even be in favor of dropping the read/write methods on Container to keep things simple. In my world you're generally either reading or writing a given dataset. Interleaving reads and writes is rare.

(It might be somewhat confusing that .read() doesn't actually read anything though, and .write() doesn't actually write anything)

One simple change would be to call the method dataset.get_writer(),

For writes, there's ton of other options like compression etc, need to think of an ergonomic way to support that (which is currently supported only in the DatasetBuilder). At the very least, it should be possible to reshape the dataset on the fly -- e.g. if you provide the data in a slice/Vec but want to store it as a 2-D array, maybe something like

let _ = dataset.write::<T>()?.shape((10, 20)).slice(&vec)?;

I really like the current DatasetBuilder setup. Options like compression, chunking, resize-ability need to be specified at dataset creation time, right? I don't see how they can be added to the WriteContainer. (e.g. http://docs.h5py.org/en/stable/high/dataset.html#creating-datasets). I'm most interested in the simple matched-dimensionality read/write cases. Would establishing that API force decisions that would impact a future 'reshape-on-the-fly' API?

[aldanor]

@pmarks Thanks for the response. I'll try to address your points below.

So to clarify, the proposal is that dataset.write::() would return Ok(WriteContainer) if there's a noop path from storage->T, and Err() otherwise. Whereas dataset.write_hard::() would return Ok(WriteContainer) if there was a noop or hard conversion available, but would return Err() if only a soft conversion was available?

Yes, that is correct (i.e., "noop"=noop, "hard"=noop|hard, "soft"=noop|hard|soft). It could also be possible to provide a way to pass conversion as an enum value (since the enum type exists anyway), like dataset.write_conv::<T>(Conversion::Soft), but I don't see much use for it anyway unless you're writing some generic crate around this crate -- since you pretty much always know what you need in advance, e.g. noop/hard/soft.

I'd even be in favor of dropping the read/write methods on Container to keep things simple. In my world you're generally either reading or writing a given dataset. Interleaving reads and writes is rare.

To think about, that's my experience too. In my many years of dealing with HDF5 in all possible languages, contexts and platforms, I can't think of a case where I had both a read and a write on neighboring lines of code. It's typically either one or the other.

One simple change would be to call the method dataset.get_writer(),

This might be one way, yea, hmm... Definitely heading into bikeshedding territory but, then I think you would need to add "read_/write_" prefixes back to its methods, like "dataset.get_reader::()?::read_arr()?" (so it reads more or less like an English sentence; kind of like with the currently proposed "dataset ... read ... array"). But then you have "read" twice and so it ends up being more verbose :) Need to ponder a bit on this.

Options like compression, chunking, resize-ability need to be specified at dataset creation time, right?

Correct. Post-creation the only two classes of options you can configure are dataset-access and data-transfer plists. Note also that it should be possible to augment the DatasetBuilder with write functionality (kind of like in h5py you can provide data=), so instead of just saying create(), you could end it with write() and it would instantly provide you with a typed write container (always assuming no-op conversion here, I guess, since you've just passed it a Rust type to instantiate it). Like: DatasetBuilder::new::<T>().chunk_auto().write().slice(&vec) or something of the sort.

I'm most interested in the simple matched-dimensionality read/write cases.

Me too. But the folks have been already asking above re: "whether datasets can be iterated to avoid loading them all into memory" etc, I'd expect people to have very different use cases so it'd be nice for the implementation to be flexible enough to suit most of them.