Tune transfer performance

S3Client runs several operations concurrently and can split large objects into parts or byte ranges. Throughput depends on the disk, network and service limits. Increasing the number of simultaneous operations puts more pressure on these resources and does not guarantee a faster transfer.

Open and save settings#

1. Open Tools > Options....
2. Select Operations for concurrency and range downloads, or General > Multipart transfers for the threshold and part size.
3. Change the relevant values.
4. Click OK to save, or Cancel to discard changes.
5. Start a new transfer and observe its progress. Transfers already prepared may retain their original splitting plan.

The values below are defaults for a new configuration; an existing configuration may use different settings. One MiB is 1048576 bytes.

Global and per-transfer concurrency#

Under Operations:

With the default settings, a multipart upload can send up to 4 parts of a file at once. Uploading 4 files this way would require up to 16 simultaneous part uploads. The global limit of 15 operations prevents them all from running at once. Other operations that read or write files on your computer share this limit, so they can reduce the number of parts uploaded at the same time.

Interface actions, such as displaying the contents of an S3 folder, have their own limit: Maximum number of concurrent operations for the GUI.. They do not count toward the first two limits in the table.

When you copy a file between two different accounts configured in S3Client and the S3 service performs the copy directly in multiple parts, those parts are copied one after another. Increasing Maximum number of active parts per multipart transfer: therefore does not speed up this type of copy.

S3 image, PDF and media previews have their own settings under Preview, including Maximum concurrent requests per preview:. See previews to adjust their limits.

Multipart threshold and part size#

Under General > Multipart transfers:

The threshold applies to local uploads and S3-to-S3 copies. Objects at or above it can use multipart when the account supports it. Service limits may require multipart regardless of the threshold. The preferred size is used to plan new parts; S3Client increases it when necessary to meet part-size and part-count limits.

In File > Accounts... > Edit... > Capabilities, set Multipart upload according to actual destination support. Server-side part copying also requires Multipart server-side copy and compatible direct copying. These capabilities do not authorize requests: initiation, upload, completion and cleanup permissions must also be available.

Smaller parts create more requests; larger parts increase the amount of data processed at once. Gzip compression changes the upload size and can change how it is split. An S3-to-S3 relay also uses memory and may use temporary files on your computer.

Range downloads#

Under Operations, Parallel download threshold: defaults to 8 MiB and accepts 1 to 1024 MiB. Files at or above this size can be downloaded using simultaneous ranges if the range limit is greater than 1 and S3Client can verify that they read the same version of the file, using its ETag or a usable version ID.

S3Client assembles the byte ranges before making the destination file available. This uses memory or a temporary file in the destination folder. If a successful service response does not match the requested byte range, S3Client can restart the download with a single request. Permission errors, inconsistent encoding and source changes remain errors.

An object reported with Content-Encoding: gzip is decompressed after reading, so its local size may exceed the transferred size. Allow space for final content and temporary files.

Observe the effects#

Compare a representative transfer before and after changing a setting, using the same files and service. Watch the Transfer column, total time, service errors, and disk, network and memory usage. Folder progress may include skipped files, so it does not measure network throughput on its own.

Adjust local concurrency when the disk is saturated, cloud concurrency when the service limits requests, and parts or ranges when a large file underuses the connection. Change one group of settings at a time to identify its effect. Detailed logging and gzip compression can also affect a measurement.