It's impossible to reliably upload a complete file

[viliam-durina]

We're trying to upload a file to GCS, and make sure the file is only published if the upload succeeds. Consider this code:

BlobInfo blobInfo = ...;
ByteBuffer buf1, buf2;
WriteChannel writer = storage.writer(blobInfo, Storage.BlobWriteOption.crc32cMatch());
try {
  writer.write(buf1);
  writer.write(buf2); // network goes down before this call
} finally {
  writer.close(); // network is back up before this call
}

The write of buf1 is successful, but before writing buf2, the network goes down. As expected, an IOException is thrown (both my buffers are 32MB). The execution proceeds to the finally clause to call close(). At this point, the network is up again. The close call finishes the file upload, and the file is published with partial contents!

However, the problem is more generic. While writing the file, any error can occur (e.g. an IO error while reading the file to upload). In this case, the close() call will also publish a partial file.

I have tested moving the close() call to the try block, so that the WriteChannel is abandoned without closing it. This seems to have the desired effect, but since the close() method is inherited from AutoCloseable, it's very unintuitive, and some code quality tools will suggest or force calling the close() method.

One possible backward-compatible fix to this problem would be to add abort() method, that will clean up the resources without finishing the upload, and after which close() will be no-op. That method could be automatically called for internal exceptions.

I'm using version 2.50.0 on Java 21.

[godwinpang]

you can either use the json api directly or raise a feature request for it to be supported in the java client lib

https://stackoverflow.com/questions/71260411/multi-part-upload-google-storage#:~:text=Currently%20Java%20Client,with%20uploadType%3Dmultipart

[BenWhitehead]

It sounds like there are multiple things at play here.

WriteChannel is ultimately backed by a Resumable Upload. This means, there are actually multiple RPCs to GCS happening over the course of the interaction with the WriteChannel.

Given two writes of 32MB (2 * 32MB = 64MB = 64,000,000 bytes being written). The following will happen:

1. when storage.writer is invoked a resumable upload will be started with GCS for the Object metadata and options provided from the BlobInfo and BlobWriteOptions.
2. when writer.write(buf1) is called
   a. 16MiB (16,777,216 bytes) will be PUT to the resumable upload from 1
   b. The remaining 15,222,784 bytes will be copied to the internal buffer
   c. return 32,000,000
3. when writer.write(buf2) is called
   a. 1,554,432 bytes will be consumed in conjunction with the existing 15,222,784 bytes from the internal buffer, these 16MiB will be PUT to the resumable upload from 1
   b. 16MiB of the remaining 30,445,568 bytes will be PUT to the resumable upload from 1
   c. The remaining 13,668,352 will be copied to the internal buffer
   d. return 32,000,000
4. when writer.close() is called
   a. The 13,668,352 from the internal buffer will be PUT to the resumable upload from 1, at the same time finalizing the resumable upload
   b. upon successful validation of all preconditions specified when the resumable upload was started, GCS will make the object live.

All PUTs are automatically retried within the budget the client has been configured with. If the retry budget is exhausted or an unretryable error is encountered, the exception will be thrown. Many SocketExceptions are automatically retried as well as a host of other possible errors. See https://cloud.google.com/storage/docs/retry-strategy for an overview and possible settings.

If a PUT fails during a write(ByteBuffer) and the retry budget is exhausted the exception will be thrown out, possibly triggering a try-with-finally invocation of writer.close(). This particular behavior is not always ideal, and is something we are looking at designing some more configurability around later this year.

Regarding the preconditions validation; a precondition like BlobWriteOption.crc32cMatch() tells GCS that the checksum of all of the bytes present in the resumable upload at finalization time MUST match the value provided in BlobInfo#getCrc32c(). From the GCS perspective, finalization can happen at any time and when it does the preconditions will be validated. If you are providing a crc32c value as part of the BlobInfo passed to storage.writer, GCS will validate it. The likelihood of a crc32c collision is not impossible, but is remote (crc32c's are unsigned 32-bit ints). We test that crc32c plumbing and validation takes place on a resumable upload, you can see the test here:

java-storage/google-cloud-storage/src/test/java/com/google/cloud/storage/it/ITObjectChecksumSupportTest.java

Lines 179 to 199 in 9bc2b14

	public void testCrc32cValidated_writer_expectFailure() {
	String blobName = generator.randomObjectName();
	BlobId blobId = BlobId.of(bucket.getName(), blobName);
	BlobInfo blobInfo = BlobInfo.newBuilder(blobId).setCrc32c(content.getCrc32cBase64()).build();
	byte[] bytes = content.concat('x');
	StorageException expected =
	assertThrows(
	StorageException.class,
	() -> {
	try (ReadableByteChannel src = Channels.newChannel(new ByteArrayInputStream(bytes));
	WriteChannel dst =
	storage.writer(
	blobInfo,
	BlobWriteOption.doesNotExist(),
	BlobWriteOption.crc32cMatch())) {
	ByteStreams.copy(src, dst);
	}
	});
	assertThat(expected.getCode()).isEqualTo(400);
	}

[viliam-durina]

We can't use the CRC option without significant code change. We're generating the contents of the blob from other data source (a kafka topic), so we don't know the CRC before opening the writer. We would need to first create the file on disk, ensure there's enough space (the size might be significant), calculate the CRC and then upload.

Is there any other option without providing the CRC beforehand? E.g. provide it just before the call to close?

[BenWhitehead]

I mentioned the crc32c option as it was included in your code snippet when opening the issue.

If crc32c isn't an option, you could use com.google.cloud.storage.Storage.BlobWriteOption#expectedObjectSize(long) if you know the number of bytes the object should be.

Retry budget configuration is covered in https://cloud.google.com/storage/docs/retry-strategy and might allow you to choose better retry settings for your deployment.

If you have access to ephemeral disk space, you could try using our BuffferToDiskThenUpload BlobWriteSession config. This will buffer pending bytes to disk and then attempt to upload the entire file in a single PUT, but picking up at the last ack'd byte if the connection is interrupted.

    try (Storage s = StorageOptions.http()
        .setBlobWriteSessionConfig(
            BlobWriteSessionConfigs.bufferToDiskThenUpload(Paths.get("/path/to/ephemeral/disk"))
        )
        .build().getService()) {
      BlobWriteSession session = s.blobWriteSession(info, BlobWriteOption.doesNotExist());

      try (WritableByteChannel w = session.open()) {
        w.write(buf1);
        w.write(buf2);
      }

      ApiFuture<BlobInfo> result = session.getResult();
      BlobInfo gen1 = result.get();
    }

[viliam-durina]

@BenWhitehead, thanks for your reply. expectedObjectSize is unusable for us for the same reason for which CRC is unusable - we're generating the file on the fly, and know neither crc nor size beforehand. And we don't have sufficient disk space locally as well.

The proposals here are basically workarounds to the core issue. However, a way to abort writing would fix it without the need for workarounds.