Consider more efficient serialization of CmdStanFit objects

[wlandau]

Is your feature request related to a problem? Please describe.

Posterior samples take up a lot of space, and RDS scales poorly. Without compression, the files get pretty big. With gzfile compression, serialization takes a lot of time.

Describe the solution you'd like

I propose a new qs-based save method analogous to the current $save_object().

Describe alternatives you've considered

If all the data in a CmdStanFit object can be expressed as a collection of data frames, maybe fst and arrow could be alternatives, but that would require multiple files per object and thus more bookkeeping.

[mitzimorris]

what proportion of processing time is spent in writing a StanFit object to RDS and what proportion of time is spent obtaining a sample from the CmdStan CSV files?

[wlandau]

I take your point that HMC is usually far more time-consuming than saving and loading samples. But downstream postprocessing tends to be quick, and I tend to iterate over it more, which means time spent reading large collections of draws is noticeable.

On reflection, if depending on qs is burdensome, what about a single method just to load all the relevant output into memory so the user can choose how to save it?

cmdstanr/R/fit.R

Lines 16 to 18 in d27994f

	self$draws()
	try(self$sampler_diagnostics(), silent = TRUE)
	try(self$init(), silent = TRUE)

[mike-lawrence]

I may be wrong, but I believe @mitzimorris wasn't talking about HMC sampling times, but the fact that at present cmdstanr has to read the samples from the CSV files that cmdstan generates, and it may be that reading time (soon to be faster) that is the primary bottleneck, at least when first accessing the data. Even so, I also think @wlandau's point is that there are use cases for re-loading the data across sessions that would be hastened by use of qs rather than rds.

[wlandau]

I may be wrong, but I believe @mitzimorris wasn't talking about HMC sampling times, but the fact that at present cmdstanr has to read the samples from the CSV files that cmdstan generates, and it may be that reading time (soon to be faster) that is the primary bottleneck, at least when first accessing the data.

Yes, my mistake. But CSV file processing is upstream of saving the object, so I still stand by my more general point.

[mike-lawrence]

(Sorry @wlandau, yes and I edited my response here after realizing my correction was indeed tangential to your point; I now see how the variability in whether folks view issues via email [where they don't see edits] vs on github is a reason for not using the edit feature here)

[jgabry]

Thanks for opening the issue @wlandau. I have no particular attachment to the RDS format, so I'm definitely open to better options.

On reflection, if depending on qs is burdensome, what about a single method just to load all the relevant output into memory so the user can choose how to save it?

I like the idea of letting the user choose how to save things and if we can do that without adding an additional dependency then that would be great. So this seems like a good approach. What do others think?

[mitzimorris]

I agree that we need good serialization - but we need to think cross-interface and therefore find serialization formats that can be accessed by Python and C++ as well as R-specific formats.

[jgabry]

Yeah I definitely like the idea of using formats that are compatible across the interfaces. I don't think that conflicts with adding a method to reading everything into memory though. Right now the user has to call multiple different methods to get everything into memory (draws(), sampler_diagnostics(), etc.), which is fine for interactive use but a pain if they just want to read everything in to save it. So a single method that reads everything into memory seems like a good idea to me. @mitzimorris Won't we need that anyway regardless of which serialization format is used? Or am I misunderstanding (totally possible!).

[mitzimorris]

how are sampler_diagnostics() different from draws()? is this implemented differently in CmdStanR than in CmdStanPy?

aren't the first 7 columns of the draws array the source for sampler_diagnostics? or is this something extra computed by CmdStanR outside of CmdStan? (feature creep!)

the fundamental problem we're working against now is the way that the underlying Stan C++ code for the algos and services send all output information to a single CSV file. for backwards compatibility, we work with what we've got.
at the same time, we should remember to keep a wholistic view of the workflow and redesign the I/O accordingly.

[jgabry]

Yeah the sampler diagnostics are the first columns of the same CSV file containing the samples, but because those columns serve a different purpose than the other columns the draws() method doesn't return those columns, just the sampler_diagnostics() method. The draws() method returns the parameters, transformed parameters, and generated quantities, and the sampler_diagnositcs() method returns the diagnostic columns. Here's the original issue with code examples:

#123

Is that not what CmdStanPy does?

[mitzimorris]

in CmdStanPy, everything is stored in a single numpy.ndarray structure

• stored column major, so all the sampler diags are contiguous in memory
• views on the array will give you sampler diags or draws, (although this isn't implemented)

initial focus for CmdStanPy was truly lightweight wrapper - it was important to get info from CSV files into a data structure that had the right memory locality properties for downstream analysis. this could be easily modified to have sampler_params and model_params (or sampler_diagnostics and draws, for uniformity) - there's a few open issues already - could be done as part of stan-dev/cmdstanpy#171

[jgabry]

Oh I see what you mean. We're currently storing the diagnostics in one array and the other draws in a another array (once they're read in from CSV), but we could switch to storing them together if that's preferable (the method would continue to return them separately). I didn't think it would matter too much but I could definitely be wrong!

Also @rok-cesnovar any thoughts on any of this?

[mitzimorris]

my concern is making sure interfaces aren't too different from one another - we must allow idioms appropriate to the language (i.e., "pythonic" vs "rthonic" - use of quotes quite intentional)