Merge pull request #181 from EmmaZhu/putblobfromurl

Add support for service side sync copy

Author: vinjiang

BreakingChanges.txt

@@ -1,3 +1,6 @@
+Tracking Breaking Changes since 0.12.0
+   - Changed parameter isServiceCopy of bool value to copyMethod which is an enumeration value in interfaces of CopyAsync and CopyDirectoryAsync.
+
 Tracking Breaking Changes since 0.10.1
    - Changed base namespace from Microsoft.WindowsAzure.Storage.DataMovement to Microsoft.Azure.Storage.DataMovement, 
      to keep align with azure storage client libraries' namespace change from Microsoft.WindowsAzure.Storage to Microsoft.Azure.Storage, assembly name is changed accordingly.

README.md

@@ -1,4 +1,4 @@
-# Microsoft Azure Storage Data Movement Library (0.12.0)
+# Microsoft Azure Storage Data Movement Library (1.0.0)
 
 The Microsoft Azure Storage Data Movement Library designed for high-performance uploading, downloading and copying Azure Storage Blob and File. This library is based on the core data movement framework that powers [AzCopy](https://azure.microsoft.com/documentation/articles/storage-use-azcopy/).
 
@@ -11,13 +11,13 @@ For more information about the Azure Storage, please visit [Microsoft Azure Stor
 
 - Blobs
     - Download/Upload/Copy Blobs.
-    - Synchronous and asynchronous copy Blobs
+    - Copy Blobs with synchronous copying, service side asynchronous copying and service side synchronous copying.
     - Concurrently transfer Blobs and Blob chunks, define number of concurrent operations
     - Download Specific Blob Snapshot
 
 - Files
 	- Download/Upload/Copy Files.
-    - Synchronous and asynchronous copy Files
+    - Copy File with synchronous copying and service side asynchronous copying.
     - Concurrently transfer Files and File ranges, define number of concurrent operations
 
 - General
@@ -120,6 +120,55 @@ var task = TransferManager.UploadAsync(
 	sourcePath, destBlob, null, context, CancellationToken.None);
 task.Wait();
 ```
+
+### Copy a blob
+
+First, include the classes you need, which is the same as the sample to Upload a blob.
+
+```csharp
+using System;
+using System.Threading;
+using Microsoft.Azure.Storage;
+using Microsoft.Azure.Storage.Blob;
+using Microsoft.Azure.Storage.DataMovement;
+```
+
+Now use the interfaces provided by Storage client lib to setup the storage contexts (find more details at [how to use Blob Storage from .NET](https://azure.microsoft.com/documentation/articles/storage-dotnet-how-to-use-blobs/)):
+
+```csharp
+string sourceStorageConnectionString = "sourceStorageConnectionString";
+CloudStorageAccount sourceAccount = CloudStorageAccount.Parse(sourceStorageConnectionString);
+CloudBlobClient sourceBlobClient = sourceAccount.CreateCloudBlobClient();
+CloudBlobContainer sourceBlobContainer = sourceBlobClient.GetContainerReference("sourcecontainer");
+CloudBlockBlob sourceBlob = sourceBlobContainer.GetBlockBlobReference("sourceBlobName");
+
+string destStorageConnectionString = "destinationStorageConnectionString";
+CloudStorageAccount destAccount = CloudStorageAccount.Parse(destStorageConnectionString);
+CloudBlobClient destBlobClient = sourceAccount.CreateCloudBlobClient();
+CloudBlobContainer destBlobContainer = sourceBlobClient.GetContainerReference("destinationcontainer");
+CloudBlockBlob destBlob = sourceBlobContainer.GetBlockBlobReference("destBlobName");
+```
+
+Once you setup the storage blob contexts, you can start to use `WindowsAzure.Storage.DataMovement.TransferManager` to copy the blob and track the copy progress:
+
+```csharp
+// Setup the number of the concurrent operations
+TransferManager.Configurations.ParallelOperations = 64;
+// Setup the transfer context and track the copy progress
+SingleTransferContext context = new SingleTransferContext();
+context.ProgressHandler = new Progress<TransferStatus>((progress) =>
+{
+	Console.WriteLine("Bytes Copied: {0}", progress.BytesTransferred);
+});
+
+// Copy a blob
+var task = TransferManager.CopyAsync(
+    sourceBlob, destBlob, CopyMethod.ServiceSideSyncCopy, null, context, CancellationToken.None);
+task.Wait();
+```
+
+DMLib supports three different copying methods: Synchronous Copy, Service Side Asynchronous Copy and Service Side Synchronous Copy. The above sample uses Service Side Synchronous Copy. See [Choose Copy Method](#choose-copy-method) for details on how to choose the copy method. 
+
 # Best Practice
 
 ### Increase .NET HTTP connections limit
@@ -200,6 +249,66 @@ The following matrix explains how the DirectoryOptions.Recursive and DirectoryOp
 
 - Default recursive option: false
 
+### Choose Copy Method
+DMLib supports three copy methods:
+- Synchronous Copy  
+  DMLib downloads data from source to memory, and uploads the data from memory to destination.
+    
+- Service Side Asynchronous Copy  
+  DMLib sends request to Azure Storage Server to start the copying, and monitors its status until the copying is completed.
+      
+- Service Side Synchronous Copy  
+  DMLib leverages [Put Block From URL](https://docs.microsoft.com/en-us/rest/api/storageservices/put-block-from-url), [Append Block From URL](https://docs.microsoft.com/en-us/rest/api/storageservices/append-block-from-url), [Put Page From URL](https://docs.microsoft.com/en-us/rest/api/storageservices/put-page-from-url) to copy Azure Storage Blobs.
+ 
+Following is suggested copy method for different scenarios:
+- From performance aspect: <br> 
+Based on Azure Storage Server's [SLA for Storage Accounts](https://azure.microsoft.com/en-us/support/legal/sla/storage/v1_5/), following are suggestions when copying performance is most important to you: 
+  - To copy between blobs or files inner storage account, Service Side Asynchronous Copy would be suggested.
+  - To copy Blobs, Service Side Synchronous Copy would be suggested.
+  - To copy Files or copy between Blobs and Files, Synchronous Copy would be suggested. To achieve best copying performance, Synchronous Copy would need a powerful machine in Azure.
+- From cost aspect, Service Side Asynchronous Copy would cost least. 
+- From data flow approach, Synchronous Copy would be well controlled. With Synchronous Copy, the data will go through the network configured by you.
+- From supported directions, Synchronous Copy and Service Side Asynchronous Copy support more directions than Service Side Synchronous Copy. See details in below table
+  
+Following table shows supported directions with different copy method.
+  <table>
+  <tr>
+    <td></td>
+    <th scope="col">Append Blob</th>
+    <th scope="col">Block Blob</th>
+    <th scope="col">Page Blob</th>
+    <th scope="col">Azure File</th>
+  </tr>
+  <tr>
+    <th scope="row">Append Blob</th>
+    <td>Synchronous Copy <br> Service Side Asynchronous Copy <br> Service Side Synchronous Copy</td>
+    <td>N/A</td>
+    <td>N/A</td>
+    <td>Synchronous Copy <br> Service Side Asynchronous Copy</td>
+  </tr>
+  <tr>
+    <th scope="row">Block Blob</th>
+    <td>N/A</td>
+    <td>Synchronous Copy <br> Service Side Asynchronous Copy <br> Service Side Synchronous Copy</td>
+    <td>N/A</td>
+    <td>Synchronous Copy <br> Service Side Asynchronous Copy</td>
+  </tr>
+  <tr>
+    <th scope="row">Page Blob</th>
+    <td>N/A</td>
+    <td>N/A</td>
+    <td>Synchronous Copy <br> Service Side Asynchronous Copy <br> Service Side Synchronous Copy</td>
+    <td>Synchronous Copy <br> Service Side Asynchronous Copy</td>
+  </tr>
+  <tr>
+    <th scope="row">File</th>
+    <td>Synchronous Copy <br> Service Side Asynchronous Copy</td>
+    <td>Synchronous Copy <br> Service Side Asynchronous Copy</td>
+    <td>Synchronous Copy <br> Service Side Asynchronous Copy</td>
+    <td>Synchronous Copy <br> Service Side Asynchronous Copy</td>
+  </tr>
+</table>
+
 # Need Help?
 Be sure to check out the Microsoft Azure [Developer Forums on MSDN](http://go.microsoft.com/fwlink/?LinkId=234489) if you have trouble with the provided code or use StackOverflow.
 

changelog.txt

@@ -1,3 +1,8 @@
+2019.08.08 Version 1.0.0
+ * Blob Service on both .Net Framework and .Net Core
+   - Added a copy method to leverage Service Side Synchronous Copy when copying blobs.
+   - Changed default block size in uploading block blob from 8MB to 4MB.
+
 2019.07.05 Version 0.12.0
  * All Services on both .Net Framework and .Net Core
    - Upgraded Microsoft.Azure.Storage.Blob from 10.0.1 to 10.0.3 

lib/Constants.cs

@@ -20,9 +20,14 @@ public static class Constants
         public const int MaxBlockSize = 100 * 1024 * 1024;
 
         /// <summary>
-        /// Default block size, 4MB.
+        /// Default block size when transferring block blob, 8MB.
         /// </summary>
-        public const int DefaultBlockSize = 4 * 1024 * 1024;
+        public const int DefaultBlockBlobBlockSize = 8 * 1024 * 1024;
+
+        /// <summary>
+        /// Default size for a chunk transferring, 4MB.
+        /// </summary>
+        public const int DefaultTransferChunkSize = 4 * 1024 * 1024;
 
         /// <summary>
         /// Default memory chunk size of memory pool, 4MB.

lib/CopyMethod.cs

@@ -0,0 +1,34 @@
+//------------------------------------------------------------------------------
+// <copyright file="CopyMethod.cs" company="Microsoft">
+//    Copyright (c) Microsoft Corporation
+// </copyright>
+//------------------------------------------------------------------------------
+
+
+namespace Microsoft.Azure.Storage.DataMovement
+{
+    /// <summary>
+    /// Enum to indicate how the copying operation is handled in DataMovement Library.
+    /// </summary>
+    public enum CopyMethod
+    {
+        /// <summary>
+        /// To download data from source to memory, and upload the data from memory to destination.
+        /// </summary>
+        SyncCopy,
+
+        /// <summary>
+        /// To send a start copy request to azure storage to let it do the copying,
+        /// and monitor the copying progress until the copy completed.
+        /// </summary>
+        ServiceSideAsyncCopy,
+
+        /// <summary>
+        /// To copy content of each chunk with with Put Block From URL, Append Block From URL or Put Page From URL.
+        /// See <c>https://docs.microsoft.com/en-us/rest/api/storageservices/put-block-from-url</c> for Put Block From URL,
+        /// <c>https://docs.microsoft.com/en-us/rest/api/storageservices/append-block-from-url</c> for Append Block From URL,
+        /// <c>https://docs.microsoft.com/en-us/rest/api/storageservices/put-page-from-url</c> for Put Page From URL for details.
+        /// </summary>
+        ServiceSideSyncCopy
+    }
+}

lib/DataMovement.csproj

@@ -85,6 +85,7 @@
     </Compile>
     <Compile Include="AssemblyInfo.cs" />
     <Compile Include="ChunkedMemoryStream.cs" />
+    <Compile Include="CopyMethod.cs" />
     <Compile Include="DirectoryListingScheduler.cs" />
     <Compile Include="Extensions\StorageCopyState.cs" />
     <Compile Include="FileNativeMethods.cs" />
@@ -115,6 +116,10 @@
     <Compile Include="TransferControllers\AsyncCopyControllers\FileAsyncCopyController.cs" />
     <Compile Include="TransferControllers\AsyncCopyControllers\AsyncCopyController.cs" />
     <Compile Include="TransferControllers\DummyTransferController.cs" />
+    <Compile Include="TransferControllers\ServerSideSyncCopyControllers\AppendBlobServiceSideSyncCopyController.cs" />
+    <Compile Include="TransferControllers\ServerSideSyncCopyControllers\BlockBlobServiceSideSyncCopyController.cs" />
+    <Compile Include="TransferControllers\ServerSideSyncCopyControllers\PageBlobServiceSideSyncCopyController.cs" />
+    <Compile Include="TransferControllers\ServerSideSyncCopyControllers\ServiceSideSyncCopyController.cs" />
     <Compile Include="TransferControllers\TransferReaders\CloudFileReader.cs" />
     <Compile Include="TransferControllers\TransferReaders\PageBlobReader.cs" />
     <Compile Include="TransferControllers\TransferReaders\RangeBasedReader.cs" />

lib/Resources.Designer.cs

@@ -269,6 +269,9 @@ MD5 in property: {2}</value>
   <data name="RestartableInfoCorruptedException" xml:space="preserve">
     <value>Failed to read restartable info from file.</value>
   </data>
+  <data name="ServiceSideSyncCopyNotSupportException" xml:space="preserve">
+    <value>Service side synchronous copying is only supported when destination is Azure Blob Storage.</value>
+  </data>
   <data name="SmallMemoryCacheSizeLimitationException" xml:space="preserve">
     <value>MaximumCacheSize cannot be less than {0}.</value>
     <comment>{0} is minimum memory cache size limitation</comment>
@@ -336,6 +339,12 @@ MD5 in property: {2}</value>
   <data name="AzureFile" xml:space="preserve">
     <value>AzureFile</value>
   </data>
+  <data name="ServiceSideSyncCopyFromFileNotSupportedException" xml:space="preserve">
+    <value>Copying from Azure File Storage with service side synchronous copying is not supported.</value>
+  </data>
+  <data name="ServiceSideSyncCopyToFileNotSupportedException" xml:space="preserve">
+    <value>Copying to Azure File Storage with service side synchronous copying is not supported.</value>
+  </data>
   <data name="SyncCopyFromUriToAzureBlobNotSupportedException" xml:space="preserve">
     <value>Copying from uri to Azure Blob Storage synchronously is not supported.</value>
   </data>

lib/Resources.resx

@@ -44,7 +44,7 @@ public class TransferConfigurations
         public TransferConfigurations()
         {
             // setup default values.
-            this.blockSize = Constants.DefaultBlockSize;
+            this.blockSize = Constants.DefaultTransferChunkSize;
             this.parallelOperations = Environment.ProcessorCount * 8;
             this.MemoryChunkSize = Constants.DefaultMemoryChunkSize;
 
@@ -107,7 +107,7 @@ public int BlockSize
                     throw new ArgumentOutOfRangeException("value", value, errorMessage);
                 }
 
-                if (value % Constants.DefaultBlockSize != 0)
+                if (value % Constants.DefaultTransferChunkSize != 0)
                 {
                     throw new ArgumentException(Resources.BlockSizeMustBeMultipleOf4MB, "value");
                 }
@@ -140,12 +140,12 @@ internal long MaximumCacheSize
 
             set
             {
-                if (value < Constants.DefaultBlockSize)
+                if (value < Constants.DefaultTransferChunkSize)
                 {
                     throw new ArgumentException(string.Format(
                         CultureInfo.CurrentCulture,
                         Resources.SmallMemoryCacheSizeLimitationException,
-                        Utils.BytesToHumanReadableSize(Constants.DefaultBlockSize)));
+                        Utils.BytesToHumanReadableSize(Constants.DefaultTransferChunkSize)));
                 }
 
                 if (0 == this.memStatus.AvailablePhysicalMemory)

lib/TransferConfigurations.cs

@@ -166,21 +166,6 @@ protected abstract Uri DestUri
             get;
         }
 
-        public static AsyncCopyController CreateAsyncCopyController(TransferScheduler transferScheduler, TransferJob transferJob, CancellationToken cancellationToken)
-        {
-            if (transferJob.Destination.Type == TransferLocationType.AzureFile)
-            {
-                return new FileAsyncCopyController(transferScheduler, transferJob, cancellationToken);
-            }
-
-            if (transferJob.Destination.Type == TransferLocationType.AzureBlob)
-            {
-                return new BlobAsyncCopyController(transferScheduler, transferJob, cancellationToken);
-            }
-
-            throw new InvalidOperationException(Resources.CanOnlyCopyToFileOrBlobException);
-        }
-
         /// <summary>
         /// Do work in the controller.
         /// A controller controls the whole transfer from source to destination,