Integrating Aspera High-Speed Transfer Server (Enterprise Server) with

SwiftStack Storage

How-To Guide: Integrating Aspera with SwiftStack

Introduction 2 Aspera + SwiftStack Architecture 2

Installation and Basic Setup 5 Configuring SwiftStack for Aspera 5 Installing Aspera High-Speed Transfer Server on Linux 10 Enabling Aspera TrapD Service 11 Choosing an API (Swift vs. S3) 11

Swift API 11 S3 API 12

Performance, Memory, and Very Large File Considerations 12 Modifying TrapD Properties File(s) 14

Configure Aspera Docroot for SwiftStack 15 Adding a New Aspera User 15 Setting the Docroot 17

Validating the Configuration 17 Aspera Desktop Client 17 SwiftStack Client 22 Cyberduck 24

Conclusion 30

1

How-To Guide: Integrating Aspera with SwiftStack

Introduction This SwiftStack and Aspera integration guide describes the steps necessary to install and configure Aspera High-Speed Transfer Server (formerly known as Aspera Enterprise Server) to use SwiftStack as a storage backend. It also outlines testing steps and client applications that can (and should) be used to validate the configuration once complete. This solution makes for a blazing fast, infinitely scalable, multi-site approach to data transfer and storage that is unrivaled by the competition. The author of this guide architected, administered, and operated Aspera as a customer for 6+ years prior to joining the SwiftStack team. As one of the first joint SwiftStack and Aspera customers, the instructions and advice in this guide come from 2+ years of experience running Aspera and SwiftStack together in production at multi-petabyte scale. In addition to detailed installation, configuration, and validation instructions, this document describes the overall architecture of an Aspera + SwiftStack solution and provides performance details that can assist in tuning for specific use cases—such as those that require transferring and storing very large files. Aspera has natively supported object storage (a.k.a. cloud storage) for many years and is one of the few applications that can integrate with SwiftStack using both the S3 API and the OpenStack Swift API—thanks to Aspera’s history with OpenStack-based public cloud providers like Rackspace, HP Cloud, and IBM SoftLayer. In fact, you can see references to some of these public cloud providers in the swift.properties file, which we will review later in this guide. We will also cover the few caveats between the APIs when using Aspera with SwiftStack so that you can make an educated decision on which is best for your use case.

Aspera + SwiftStack Architecture Aspera and SwiftStack can be used together in two different ways that are not mutually exclusive. The use case typically dictates the overall architecture of both solutions. Aspera + SwiftStack Use Cases:

● Traditional transfer acceleration: Aspera can be used to allow external parties to securely transfer directly to/from SwiftStack—usually via the public Internet.

● Internal data-mover: Aspera can be used to move data between your existing NAS/SAN and SwiftStack.

2

How-To Guide: Integrating Aspera with SwiftStack

The first and likely most popular method is to use Aspera to send and receive transfers via the public Internet (or WAN link) with SwiftStack as the backend storage solution used by Aspera. SwiftStack can be deployed on-premises as single- or multi-region private cloud storage and extend to a hybrid cloud by including public cloud storage as well.

Image 1: Aspera sends and receives transfers via the public Internet to an on-premises, multi-region SwiftStack cluster.

Image 2: Aspera sends and receives transfers via the public Internet to a hybrid cloud storage environment. The second method is to use Aspera as a dedicated “data mover” on the LAN between “tier 1” NAS/SAN storage and “tier 2” SwiftStack private cloud storage.

Image 3: Aspera moves data on the LAN between storage tiers.

3

How-To Guide: Integrating Aspera with SwiftStack

Because these use cases are not mutually exclusive, the third image shows both strategies in use at the same time.

Image 4: Aspera handles data transfers in and out of the environment as well as between storage tiers. When Aspera writes to SwiftStack (and most S3 API compatible object stores), it utilizes S3 API MPU (multi-part upload) functionality for files above a certain size threshold (64MB by default). This threshold can be configured in TrapD’s ‘<api>.properties’ files to optimize performance or tune for different use cases. Data is sent/written to SwiftStack via Aspera’s TrapD service in the form of HTTP PUT requests and read from SwiftStack using equivalent HTTP GET requests. SwiftStack durably stores data using either replication or erasure coding across one or more physical locations (private cloud) and optional public locations such as AWS and GCP. Unlike traditional NAS/SAN file storage where Aspera performs I/O directly to and from disk, SwiftStack (and all object storage) requires a different strategy: Aspera’s TrapD service performs all initial transfer I/O to and from RAM, where it coalesces transferred data “blocks” until a specified threshold is met and a PUT is performed. We will review these tuning thresholds and their implications on hardware requirements later in this guide. SwiftStack and Aspera make for an ideal transfer and storage solution that is highly secure with built-in disaster recovery—all while avoiding vendor lock-in and at a much lower cost than traditional NAS or SAN solutions.

4

How-To Guide: Integrating Aspera with SwiftStack

Installation and Basic Setup The following steps provide an example of how to configure Aspera and SwiftStack together.

Configuring SwiftStack for Aspera Note 1: Your SwiftStack cluster should have a properly signed SSL certificate installed prior to performing this integration. Note 2: You can skip this section and proceed to “Installing Aspera” or “Configuring Aspera” if you want to use an existing SwiftStack account and container (bucket). If you do want to use an existing account and container, please make sure you have the credentials and container name ready for steps later in the guide.

1. Use the SwiftStack Controller to create a user account for Aspera. In this example, we will name the account simply ‘aspera’.

a. Click on the ‘Users’ button under the ‘Clusters’ tab of the controller.

5

How-To Guide: Integrating Aspera with SwiftStack

b. Click ‘Create New User’

6

How-To Guide: Integrating Aspera with SwiftStack

c. Provide an account username and password and click ‘Submit’.

7

How-To Guide: Integrating Aspera with SwiftStack

d. After creating the user, click ‘Push Users to Cluster’.

e. When ‘Job Finished’ appears, the new user is available on the SwiftStack cluster.

f. After successfully creating and pushing the new user, navigate back to the ‘Users’ page (step a), click ‘Show S3 API Key’ and note it for later.

2. Create a container (bucket) within the new ‘aspera’ SwiftStack account for Aspera to use.

a. In this example, we will use the SwiftStack Client (available free from https://www.swiftstack.com/downloads) and the credentials we just created to create a new container (bucket) for Aspera called ‘asperabucket’.

8

How-To Guide: Integrating Aspera with SwiftStack

Note: If you don’t know which storage policy to select for the new container, please consult with your SwiftStack administrator or reach out to the SwiftStack team for assistance.

b. For help using the SwiftStack Client, please refer to our demo videos at: https://swiftstack.com/product/demos

9

How-To Guide: Integrating Aspera with SwiftStack

Installing Aspera High-Speed Transfer Server on Linux Note: If you already have a working install of Aspera High-Speed Transfer Server (or Enterprise Server) on Linux, please skip this step and proceed to “Enabling Aspera TrapD Service.”

1. Download the latest installer for your Linux distro of choice from https://downloads.asperasoft.com/en/downloads/1 and copy it to the machine that will become your Aspera server. In this example, we will be using CentOS 7, so we will download the .rpm file and copy it to /tmp/ on the new Aspera server. We will be following the installation procedure outlined here: https://downloads.asperasoft.com/en/documentation/1

2. Install from the .rpm file: $ rpm -ivh /tmp/ibm-aspera-hsts-<version>-linux-64.rpm

3. Resolve any dependencies that arise (as per Aspera’s documentation). We had to run the following, but your results may vary based on Linux distro and installation options:

$ yum install perl $ yum install perl-Digest-MD5 $ yum install perl-Data-Dumper

4. Install Aspera license and confirm it validates: $ vi /opt/aspera/etc/aspera-license $ ascp -A

5. Configure Aspera-recommended defaults: $ asconfigurator -x "set_node_data;authorization_transfer_in_value,deny" $ asconfigurator -x "set_node_data;authorization_transfer_out_value,deny" $ asconfigurator -x "set_node_data;token_encryption_key,<your_token>" $ asconfigurator -x "set_node_data;absolute,/null/" Note: For security reasons, the above settings disable transfers by default so that the ability to transfer must be explicitly defined for every user. We will do this in a later step when we create new Aspera users.

10

How-To Guide: Integrating Aspera with SwiftStack

Enabling Aspera TrapD Service Note: If you have already enabled TrapD to interface with another object storage system such as AWS S3 or Google Cloud Storage, you can skip this step and proceed to “Editing TrapD Properties Files.”

1. Run the following command to enable Aspera’s TrapD service: $ /opt/aspera/bin/astrap-config.sh enable

(yes, it’s that easy!) Aspera Documentation: https://download.asperasoft.com/download/docs/entsrv/3.9.1/es_admin_linux/webhelp/index.html#dita-trap/trap_intro.html

Choosing an API (Swift vs. S3) As discussed in the introduction, Aspera uses a service called TrapD to support interfacing with object storage. TrapD supports both the Swift API and the S3 API as well as several other proprietary object storage protocols such as the one used by Microsoft’s Azure Blob Storage. The primary difference between Aspera’s implementation of the Swift API and the S3 API is how segments are stored for large file uploads—which mean anything over 64MB by default.

Swift API When using the Swift API, Aspera stores large object segments (also known as SLO segments) in a “hidden folder” called ‘.aspera-segments’ within the same container as all of the small objects and large object manifests. For example, uploading the 640MB file “640MB.file” to a bucket called ‘asperabucket’ will result in the creation of the SLO manifest object ‘asperabucket/640MB.file’ and 10x 64MB segment objects such as ‘asperabucket/.aspera-segments/640MB.file/segment00001,’ etc. Pros: This strategy means that the segments for large objects will always be stored using the same protection policy as all other objects (and large object manifests) in the container. This can avoid issues that arise when the default protection policy for the SwiftStack cluster differs from the

11

How-To Guide: Integrating Aspera with SwiftStack

protection policy desired to be used for Aspera. See below “cons” section for the S3 API for more detail. Cons: This strategy means that the segments are visible from within the Aspera application and to other applications interfacing with the same bucket, which can lead to confusion among users if they don’t understand why these segments exist. Storing segments in this way can also lead to issues with storage management software solutions that typically assume large object segments are stored in separate containers and/or entirely hidden from view.

S3 API When using the S3 API, Aspera stores large object segments (also known as MPU segments) in a separate bucket/container that follows the ‘<bucketname>+segments’ syntax. For example, uploading the 640MB file “640MB.file” to a bucket called ‘asperabucket’ will result in the creation of the object manifest ‘asperabucket/640MB.file’ and 10x 64MB segments in a separate bucket ‘asperabucket+segments,’ such as ‘asperabucket+segments/640MB.file/segment001,’ etc. Pros: This strategy keeps the primary bucket clean of the large number of segment objects and avoids user confusion related to the presence of segments. The S3 API is also Aspera’s most widely used object storage integration so has likely seen the most development. Cons: As briefly discussed above, storing object segments in a separate bucket can lead to surprises in the case where the default protection policy for the SwiftStack cluster differs from the protection policy desired for Aspera data. In this case, the ‘<bucketname>+segments’ container must be pre-created using the desired protection policy, which will prevent it from inheriting the default protection policy when it would otherwise be created by the s3api middleware.

Performance, Memory, and Very Large File Considerations When pushing performance limits, we recommend running Aspera on a physical (bare-metal) machine with dual socket 3Ghz+ processors, at least 128GB of RAM, and ample network connectivity to support your desired level of throughput (at least 2x10GbE). As briefly discussed earlier in this guide, Aspera is particularly memory-hungry when interacting with SwiftStack, because the TrapD service performs all I/O to and from memory, where it then coalesces transferred data until it reaches the specified segment size (64MB by default) before uploading to SwiftStack. This means that memory consumption is impacted by three variables:

1) Number of simultaneous transfers 2) Specified segment size for large file uploads

12

How-To Guide: Integrating Aspera with SwiftStack

3) Performance of object storage backend

The higher the number of simultaneous transfers, the more memory is consumed. The larger the segment size, the more memory is consumed. The slower the object storage backend, the more memory is consumed. Thankfully, SwiftStack is extremely fast! At a minimum, memory consumption will always follow: (number of simultaneous transfers) * (segment size) This is because Aspera waits for the segment size to be met before it starts a PUT to SwiftStack and only deletes the coalesced segment from memory after the PUT is complete. This means that the Aspera server’s throughput to the object storage backend must always be able to exceed the incoming client throughput, otherwise the Aspera server will run out of memory and eventually crash. Ample memory will allow the server to support many simultaneous transfers at maximum speed! On a more detailed level, actual memory consumption will always be slightly higher than the above, because Aspera is also coalescing new segments from incoming data at the same time as it is uploading segments that have reached the specified size. In a best case scenario, memory consumption will never exceed 2x the above equation, because the throughput of the object store will always be higher than the throughput the Aspera clients can achieve. For reference, the parameters that control segment size are: (more detail in next section)

aspera.transfer.max-single-put` `aspera.transfer.upload.part-size` `aspera.swift.session.upload.multipart.limit` (if using Swift API)

Regarding very large files: Like AWS S3, SwiftStack limits the total number of object segments for a single S3 API MPU (or Swift API SLO) to 10,000. When combined with Aspera’s default segment size of 64MB, this allows writing files/objects up to 640,000MB—or ~625GB. To support files larger than ~625GB, Aspera will automatically increase the segment size to a range necessary to remain under the 10,000 segment limit. This behavior is described in the ‘# Multipart” section of the TrapD properties files discussed in the next section. For use cases that require writing very large files (1TB+), we recommend increasing the defaults mentioned above to the 128MB-4GB range to avoid storing a needlessly large number of segments on SwiftStack. A reasonable guide is as follows:

13

How-To Guide: Integrating Aspera with SwiftStack

Source File Size Recommended Value for `aspera.transfer.upload.part-size`

400GB - 1TB 128MB

1TB - 5TB 1GB

5TB - 10TB 2GB

10TB+ 4GB

Modifying TrapD Properties File(s) In this section, we will modify the swift.properties file and s3.properties file to interface with SwiftStack storage. If you only want to use one API, you only need to modify the file for that API.

1. For Swift API, open /opt/aspera/etc/trapd/swift.properties in your favorite text editor, and make the following changes. If you only want to use the S3 API, please skip to Step 2 below.

a. Uncomment and modify the following lines to match:

aspera.swift.endpoint.auth-path=/auth/v1.0 aspera.swift.https-only=true

Optional performance parameters:

aspera.swift.session.upload.multipart.limit=1GB aspera.transfer.max-single-put=1GB aspera.transfer.upload.part-size=1GB

2. For S3 API, open /opt/aspera/etc/trapd/s3.properties in your favorite text editor, and make the following changes.

a. Uncomment and modify the following lines to match:

s3service.https-only=true s3service.s3-endpoint=<your cluster’s S3 API URL>

14

How-To Guide: Integrating Aspera with SwiftStack

s3service.disable-dns-buckets=true s3service.default-bucket-location=<your cluster’s S3 API region>

Optional performance parameters:

aspera.transfer.max-single-put=1GB aspera.transfer.upload.part-size=1GB

3. Restart services for changes to take effect. Note: This will interrupt any in-progress transfers.

a. $ service asperanoded restart $ service asperacentral restart $ service asperatrapd restart

Configure Aspera Docroot for SwiftStack In this section, we will review the process for creating a new Aspera user as well as for editing the docroot of an existing Aspera user. We will create two users—one for the Swift API and one for the S3 API. This is for demonstration purposes only and is not necessary unless you would like to test the differences for yourself. We will also configure appropriate transfer permissions.

Adding a New Aspera User Note: If you are working from an existing install and/or would like to change the docroot for an existing Aspera user, please skip to the next section. In this example, we will be using traditional SSH authentication with Aspera. If—for whatever reason—you need or would prefer to use Aspera’s newer internal authentication method(s), please do so on your own and then skip to Step 3 below.

1. Create a new Linux user, assign a password, and restrict to aspshell. a. Swift API:

(skip this if you want to use S3 API only) $ useradd swiftstack-asp-swift $ passwd swiftstack-asp-swift $ usermod -s /bin/aspshell swiftstack-asp-swift

15

How-To Guide: Integrating Aspera with SwiftStack

b. S3 API:

(skip this if you want to use Swift API only) $ useradd swiftstack-asp-s3api $ passwd swiftstack-asp-s3api $ usermod -s /bin/aspshell swiftstack-asp-s3api

2. Configure Aspera permissions for new user(s).

a. Swift API: (skip this if you want to use S3 API only) $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-swift;read_allowed,true" $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-swift;write_allowed,true" $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-swift;dir_allowed,true" $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-swift;authorization_transfer_in_value,allow" $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-swift;authorization_transfer_out_value,allow"

b. S3 API: (skip this if you want to use Swift API only) $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-s3api;read_allowed,true" $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-s3api;write_allowed,true" $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-s3api;dir_allowed,true" $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-s3api;authorization_transfer_in_value,allow" $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-s3api;authorization_transfer_out_value,allow"

16

How-To Guide: Integrating Aspera with SwiftStack

Setting the Docroot Configure the docroot (top level “directory”) for the new or existing Aspera user via asconfigurator.

Swift API $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-swift;absolute,swift://aspera:aspera123456789aspera@sestorage.swiftstack.tech/asperabucket"

S3 API $ asconfigurator -x "set_user_data;user_name,swiftstack-asp-s3api;absolute,s3://aspera:dc64d1222079e556fd3b36b6bcae370a@sestorage.swiftstack.tech/asperabucket"

TrapD docroots use the following syntax: <protocol>://<user>:<password-or-s3key>@<endpoint_URL>/<bucket> See: https://download.asperasoft.com/download/docs/entsrv/3.9.1/es_admin_linux/webhelp/index.html#dita-trap/object_storage_docroots.html

Validating the Configuration In this final yet important section, we will test to make sure that everything is working properly using a trio of desktop clients.

Aspera Desktop Client

1. Use the Aspera Desktop Client to verify that you can connect to the Aspera server and write to the SwiftStack endpoint via the API of your choice. In this example, we will verify both the S3 API and the Swift API.

a. If you don’t already have the Aspera Desktop Client, it is available free for all operating systems here: https://downloads.asperasoft.com/en/downloads/2 In this example, we will be using the latest version for “Mac OS X Intel.”

17

How-To Guide: Integrating Aspera with SwiftStack

b. Once installed, configure a new connection to your High-Speed Transfer Server.

S3 API:

Swift API:

18

How-To Guide: Integrating Aspera with SwiftStack

c. Connect to “SwiftStack via S3 API,” and perform a test upload. In this example,

we will be using video footage from a DJI drone. Starting the transfer:

In progress:

19

How-To Guide: Integrating Aspera with SwiftStack

Complete:

2. After successfully uploading via the API of your choice, test a download using either the same API or the other API. In this example, we will test a download via the Swift API by downloading the same file as we uploaded via the S3 API.

a. Connect to other (or same) endpoint:

20

How-To Guide: Integrating Aspera with SwiftStack

b. Initiate a download:

c. Confirm the download completed successfully:

21

How-To Guide: Integrating Aspera with SwiftStack

SwiftStack Client Now that we have confirmed Aspera can successfully upload and download from SwiftStack, let’s verify that we can see and interact with this data in other non-Aspera applications. Let’s first verify that we can download this data via the SwiftStack Client, and then we’ll upload data via SwiftStack Client and download via Aspera.

1) Configure the SwiftStack Client for your Aspera account and bucket:

2) Confirm that you can see the file that was uploaded by Aspera:

22

How-To Guide: Integrating Aspera with SwiftStack

3) Download the file that was uploaded via Aspera:

4) Upload a different file via SwiftStack Client:

23

How-To Guide: Integrating Aspera with SwiftStack

5) Verify that both objects now appear in SwiftStack Client:

6) Download the new object that was written via SwiftStack Client using Aspera:

Cyberduck Let’s repeat via Cyberduck to demonstrate true multi-application portability.

1) Download and install Cyberduck if you do not already have it installed: https://cyberduck.io/download/

24

How-To Guide: Integrating Aspera with SwiftStack

2) Configure via S3 API using credentials from the docroot examples earlier in this guide:

3) Confirm that you can see the objects uploaded by Aspera and SwiftStack Client:

25

How-To Guide: Integrating Aspera with SwiftStack

4) Upload a new file via Cyberduck:

(in progress)

26

How-To Guide: Integrating Aspera with SwiftStack

(complete)

27

How-To Guide: Integrating Aspera with SwiftStack

5) Verify that you can see the new object via SwiftStack Client:

6) Download the new object via Aspera, and confirm that it completes:

(in progress)

28

How-To Guide: Integrating Aspera with SwiftStack

(complete)

We can see and interact with the same data at the same time using two object storage APIs and with different applications!

29

How-To Guide: Integrating Aspera with SwiftStack

Conclusion Aspera and SwiftStack together make for a blazing fast, infinitely scalable, multi-site approach to data transfer and storage that is unrivaled by the competition. Both Aspera and SwiftStack are software-only solutions that allow you to avoid vendor lock-in by writing data in native format and running on any hardware platform. Thanks to Aspera’s optimized ‘fasp’ protocol and TrapD service and to SwiftStack’s cloud-native architecture, extreme levels of performance can be achieved at a much lower cost than traditional NAS or SAN solutions while achieving much higher levels of reliability and durability. Both solutions are also security-focused—allowing for data encryption in flight and at rest, so you can proceed with confidence knowing that your data is secure from prying eyes as well as infrastructure failure. Please reach out to the SwiftStack team with any questions or to schedule a demo!

30