Async function to return stream from of `into_stream` methods.

[roeap]

In one of our projects, we need to implement an ObjectStore trait form another crate backed by Azure storage account.
Conceptually this looks something like:

pub type FileMetaStream =
    Pin<Box<dyn Stream<Item = Result<FileMeta>> + Send + Sync + 'static>>;

pub struct AzureFileSystem {
    client: DataLakeClient,
}

fn convert<T>(response: ListPathsResponse) -> FileMetaStream {
    todo!()
}

impl ObjectStore for AzureFileSystem {
    async fn list_file(&self, prefix: &str) -> DFAccessResult<FileMetaStream> {
         let (file_system, prefix) = match prefix.split_once("/") {
            Some((file_system, prefix)) => (file_system.to_owned(), prefix),
            None => (prefix.to_owned(), ""),
        };

        let stream = self
            .client().
            .clone()
            .into_file_system_client(file_system)
            .list_paths()
            .directory(prefix)
            .into_stream()
            .flat_map(|f| convert(f.unwrap()));

        Ok(Box::pin(stream))
    }
}

Here the compiler complains, that the returned future does not implement Sync. To make sure, I experimented with some other operations that implement into_stream from the cosmos crate, which seems to display similar behaviour.

Trying to debug this I went through several permutations of further constraining types on Pageable etc with Send and Sync, but to no avail.

Any guidance how to further debug this, or where to look for a mitigation would wonderful :).

@rylev @ctaggart

[rylev]

Pageable should probably enforce that it is Send + Sync. It can do this by enforcing that T, E, and F in its definition are Send + Sync. You might run into issues though as I'm not sure all of our stream implementations are in fact thread safe, so you might get additional compiler errors. I'll try to take a deeper look ASAP.

cc @yoshuawuyts

[avranju]

I was kind of blocked on this issue and @rylev's suggestion seemed simple enough and took a shot at adding the additional trait constraints here - avranju@18ebbfc. It doesn't seem to have broken anything and cargo test seems to pass in all crates (checked that since pageable.rs is in core) and my app seems to build fine with this change in place. If this looks good I am happy to turn it into a PR.

[cataggar]

@avranju, yes, please submit a PR.

[avranju]

Ack. #784

[roeap]

tested with latest release and its fixed now :)

[zd4y]

Hello, I have a similar issue with Pageable<GetBlobResponse, azure_core::Error>, which doesn't implement Sync. How can I wrap that stream in something similar to FileMetaStream shown in the example?