NOTE: Relativity.Transfer.Client versions 7.4.10 and below will stop functioning on January 31, 2024, due to the migration of Relativity data transfer products to cloud-based solutions.

Check out Relativity.Transfer.SDK (NuGet), a successor of this library. It works with all types of fileshares, it's faster and simpler in use!

Since it is going to be a breaking change you will need to update your code and integrate with the Relativity.Transfer.SDK, version 1.0.0+

On December 5th, 2022 we have released the first version (1.0.3) including:

• Upload a single file or a directory to the RelativityOne fileshare
• Track data transfer progress (including files that were transferred, skipped or failed, with their source and destination path)
• Recover transfer from previous failure or interruption
• Real time information about succeeded, failed and skipped items, including exact path and the reason of error
• Cross platform compatibility - you can transfer data from Windows, Linux and macOS
• Secured transfer using HTTPS, only single TCP 443 is required

Q1 2023 you can expect additional functionalities like:

• Download a single file or a directory from the RelativityOne fileshare
• File retry policy
• File exclusion policy
• File override policy
• File custom attributes policy (used to set custom metadata on target file)

You can use the Transfer API (TAPI) to build application components that connect to Relativity and stream data from external sources into Relativity storage using different transfer protocols, for example, SMB or Aspera. You can also stream data from Relativity. The API enables optimized data transfer with extensible client architecture and event model using Relativity authentication and logging. For example, you can use the Transfer API to develop an application that loads case data into Relativity for subsequent processing. Unlike the Import API, TAPI doesn't create Relativity objects associated with the data, for example, documents and RDOs.

The TAPI includes the following core features:

• Async/await design
• Thread-safe
• Transfers in a single request or by a request job
• Massive file counts and low resources
• Cancellation using CancellationToken
• Progress using a context object
• Diagnostics (for example, compatibility and connection checks)
• Relativity user authentication
• Built-in RCC package/transfer support
• Logging with the Relativity logging framework and Serilog

We are providing a sample solution to help you get started developing your own transfer applications.

• Bluestem release for RelativityOne or on-premises Relativity
• .NET 4.6.2
• Visual C++ 2010 x86 Runtime
• Intel 2Ghz (2-4 cores is recommended)
• 8GB RAM (32GB of RAM is recommended)

Note: The Visual C++ runtime is required for Open SSL and Aspera transfer client*

As of this writing, TAPI is now integrated within the following components and applications:

• Remote Desktop Client
• Import API
• Relativity One Staging Explorer

The transfer API uses MEF (Managed Extensibility Framework) design to search and construct clients. Relativity supports the following clients:

• Aspera
• File share

Aspera transfer mode requires access to configured Aspera service. File share transfer mode requires access to locations involved in upload or download for example via vpn.

Long path support has been added in TAPI. Previous versions of TAPI had a Windows-defined maximum transfer path limit of 260 characters due to limitations with Microsoft.NET System.IO API calls. In addition to limiting the path length in the CLI, this limitation also had consequences for products that use TAPI (such as the RDC and ROSE), where attempting to transfer any paths over this 260 character limit would result in a transfer failure. This limitation existed regardless of the transfer client used.

The maximum path length now depends on the chosen transfer client. These limits apply for both the source and full target path lengths.

If a direct file share transfer is used, there is effectively no limit to the path length that can be performed. When using the Aspera transfer client, the maximum path length is now 470 due to limitations with the Aspera API. If a user specifies a source path or target path that is longer than the maximum supported path length, a fatal PathTooLongException will be thrown and the source file won't be transferred.

As part of these updates, a GlobalSetting variable has been added to adjust the behavior when a path that is too long for the chosen client to transfer is found during enumeration. This setting, called SkipTooLongPaths, is a boolean value. If true, any paths longer than the client supported maximum will be classified as an Error Path, and won't be transferred. However, enumeration and the transfer of all other valid paths will complete as part of the transfer job. If false, the enumeration will throw a fatal PathTooLongException upon encountering an invalid path length, and the transfer will fail. No files will be transferred in this situation.

The Sample.sln solution is an out-of-the-box template for developing your own custom transfer applications and demonstrates Aspera and file share API usage.

Prerequisites for running the solution:

• Visual Studio 2015, 2017 or 2019
• A Relativity instance that you can connect to
• Valid Relativity credentials

Verify the solution builds successfully. Even though the application performs no real work yet, debug to ensure the application terminates with a zero exit code.

Ensure the following 5 settings found in app.config are updated with valid values:

• TransferMode
• RelativityUrl
• RelativityUserName
• RelativityPassword
• WorkspaceId

TransferMode valid values are "Aspera" and "Fileshare". Transfer will be executed in specified mode.

The next sections discuss Aspera and file share TAPI features.

• Object model
• Demo

The next several sections highlight the TAPI object model.

• Cancellation
• InitializeGlobalSettings
• CreateTransferLog
• CreateRelativityTransferHost
• CreateTransferClient
• Subscribe to transfer events

The use of cancellation token is strongly recommended with potentially long-running transfer operations. The sample wraps a CancellationTokenSource object within a using block, assigns the CancellationToken object to a variable, and passes the variable to all asynchronous methods.

using (CancellationTokenSource cancellationTokenSource = new CancellationTokenSource())
{
    CancellationToken token = cancellationTokenSource.Token;
    ...
}

The InitializeGlobalSettings() method is responsible for configuring the GlobalSettings.

The CreateTransferLog() method uses a Relativity Logging XML configuration file to create the ITransferLog instance used to log transfer details, warnings, and errors. It's entirely possible for API users to create an ITransferLog derived class object and use virtually any logging framework; however, the RelativityTransferLog class object is provided to simplify integration with Relativity Logging. The LogConfig.xml is designed to write all entries to a rolling log file within the user profile %TEMP% directory and a local SEQ log server.

The CreateRelativityTransferHost() method defines a RelativityConnectionInfo object to specify the Relativity URL, credentials, and optional workspace artifact ID. The URL and credentials are supplied to all HTTP/REST endpoints and TAPI supports both basic authentication and OAuth2 bearer tokens. For more information about Relativity OAuth2 clients, see Relativity Documentation Site. Once constructed, the RelativityConnectionInfo object is passed to the RelativityTransferHost constructor.

If a workspace artifact is specified within the RelativityConnectionInfo object, the CreateClientAsync() method is designed to query the workspace, determine which transfer clients are supported (Aspera or file share), and choose the optimal client. If a client is specified within the ClientConfiguration object, the CreateClient() method explicitly instructs TAPI to construct a certain type of client. There may be circumstances where direct access to the file share is guaranteed and the Client will always be your best transfer option. For more information, see Dynamic Transfer Client and ITransferClient.

Since transfer events are used for both upload and download operations, the demo wraps this within the CreateTransferContext() method to construct the TransferContext object and write event details to the console. This object is used to decouple the event logic of the transfer - for example, progress and statistics - from the host and the client.

Real-world applications typically involve large or even massive datasets that not only require better transfer request management but provide real-time data rate, progress, and time remaining to their users.

For first time executions, Windows may popup a Windows Defender window like the one below. If this is presented, click the "Allow access" button.

For this demo, the approach is as follows:

• Create an Aspera or file share specific TAPI client
• Specify a target file share
• Create an upload transfer job request and job
• Search for the local dataset and add the local transfer paths to the job
• Await completion and display the results
• Create a download transfer job request and job
• Add the remote transfer paths to the job
• Change the data rate, await completion, and display the results

The CreateClientConfiguration() method is responsible for creating and configuring the ClientConfiguration object. This object inherits all of the configurable properties found within ClientConfiguration, adds numerous Aspera specific transfer properties if Aspera client selected, and assigns Aspera or Fileshare to the the Client property. This value is later evaluated by the CreateClientAsync() method to construct the specified TAPI client.

Using a workspace to drive the selected file share is convenient but doesn't meet all workflow requirements. For example, consider a data migration application to move files from on-premise to RelativityOne. In this scenario, the target workspace may not even exist; however, the migration operator knows precisely which file share should be used. For scenarios like these, the File Storage Search API is provided.

Note: You must be an admin to retrieve file shares from the instance. See Targeting file shares for more details.

Note: The GetRelativityFileShare() method supports retrieving file shares by artifact, UNC path, logical number, and name.

Once the file share is retrieved by the GetFileShareAsync() method, the object is simply assigned to the TargetFileShare property found within the ClientConfiguration object.

Data transfer workflows often involve large datasets stored on network servers or other enterprise storage devices. In many cases, the data transfer operator would like to transfer all of the files contained within a specified path. It's achieved with enumerators, created with EnumerationBuilder class. Enumerators supports features like:

• reporting current statistics,
• batching (dividing transfer on smaller chunks),
• filtering files and directories.

For large datasets (IE 1M+), it's recommended to serialize the results to disk and batching the results in smaller chunks. Since the test dataset is small, the enumeration option is used. Enumeration returns the IEnumerable of TransferPath objects and uses the lambda expression to report useful statistics.

When defining a TransferRequest object to support transfer jobs, TransferPath objects are never added to the request as this responsibility is handled by the transfer job. Among the available overloads, the ForUploadJob() and ForDownloadJob() methods accept a target path and TransferContext object.

The transfer client is used to construct a new transfer job where TransferPath objects can be added at any point in time. It's understood that files are immediately transferred in the order that they've been added to the job. The ITransferJob instance is wrapped in a using block so that all resources are properly disposed.

One of the other advantages with using a job is that it provides the API caller an object to perform job-specific operations like increasing or decreasing the data rate. Because not all TAPI client support this feature, a convenient IsDataRateChangeSupported property is provided by the ITransferJob object.

Start or debug the project and ensure 5 files are successfully uploaded/downloaded and the application terminates with a zero exit code.

The following sections provide detailed reference for the Transfer API operations illustrated by the sample program above.

The next sections cover TAPI usage including:

• RelativityConnectionInfo
• RelativityTransferHost
• ClientConfiguration
• AsperaClientConfiguration
• ITransferClient
• Dynamic transfer client
• ITransferClientStrategy
• Client support check
• Connection check
• IRemotePathResolver
• IRetryStrategy
• TransferPath
• ITransferRequest
• ITransferResult
• TransferStatus
• ITransferJob
• Transfer via request
• Transfer via job
• Workspaces and the default file share
• Targeting file shares
• Local and remote enumeration
• Change job data rate
• Transfer events and statistics
• Transfer application performance monitoring and metrics
• Error handling and ITransferIssue
• GlobalSettings
• Logging
• Relativity version check
• DateTime object values
• Binding redirect for Json.NET
• Packaging and RCC package library

The first thing you must do is construct a RelativityConnectionInfo object, which requires the following:

Note: The workspace artifact identifier can be set to Workspace.AdminWorkspaceId if the workspace is unknown or the transfer is manually configured. See Admin Workspace and Targeting file shares for more details.

The following example uses basic username/password credentials.

// This will eventually use the specified workspace to auto-configure the transfer.
const int WorkspaceId = 111111;
var connectionInfo = new RelativityConnectionInfo(
    new Uri("http://localhost/Relativity"),
    new BasicAuthenticationCredential("relativity.admin@relativity.com", "MyUbreakablePassword777!"),
    WorkspaceId);

When using an OAUTH2 client to authenticate, the bearer token is provided instead.

const int WorkspaceId = 111111;
// This will eventually use the specified workspace to auto-configure the transfer.
var connectionInfo = new RelativityConnectionInfo(
    new Uri("http://localhost/Relativity"),
    new BearerTokenCredential(bearerToken),
    WorkspaceId);

When manually configuring the transfer, do not pass the workspace artifact. In this scenario, the Admin Workspace is implicitly specified.

// This won't auto-configure the transfer.
var connectionInfo = new RelativityConnectionInfo(
    new Uri("http://localhost/Relativity"),
    new BasicAuthenticationCredential("relativity.admin@relativity.com", "MyUbreakablePassword777!"));

Given the RelativityConnectionInfo object, the RelativityTransferHost object is then constructed. This object implements IDisposable to manage object life-cycles and should employ a using block. This object has several key responsibilities including:

• Defines which Relativity instance is used for all transfer requests
• Use CreateClient to construct a specific ITransferClient object
• Use CreateClientAsync to dynamically construct the best ITransferClient object
• Use GetWorkspaceAsync to retrieve a workspace object that includes the default file share
• Use CreateFileStorageSearch to retrieve all file share objects and target a specific file share
• Use VersionCheckAsync to perform a Relativity version check

using (IRelativityTransferHost host = new RelativityTransferHost(connectionInfo))
{ }

Note: This object implements the IDisposable interface to ensure the life-cycle is managed properly.

Before you can create a client, you have to provide a ClientConfiguration instance. If you know which client you would like to construct, choose a strongly-typed class object that derives from ClientConfiguration. The vast majority of settings contained within this class object are supported by all clients unless otherwise specified.

Note: API users are strongly encouraged to set the TransferLogDirectory because clients that support this feature typically write more detailed diagnostic information into their custom log files.

The Aspera transfer engine defines a large number of properties to customize the transfer request. As a result, the AsperaClientConfiguration class object is more extensive compared to any client and special care must be taken when changing or deviating from some of the default values.

The API caller uses the transfer host to construct the appropriate transfer client. This object implements IDisposable to manage client specific object life-cycles. MEF is used to dynamically construct the appropriate instance.

// I need an Aspera client.
using (ITransferClient client = host.CreateClient(new AsperaClientConfiguration()))
{ }

// I need a file share client.
using (ITransferClient client = host.CreateClient(new FileShareClientConfiguration()))
{ }

The construction can also be driven by a IDictionary<string, string> containing name/value pairs. This is ideal in situations where the decision on which client to construct and how to configure is dynamically driven.

Dictionary<string, string> existingProperties;
using (ITransferClient client = host.CreateClient(new ClientConfiguration(existingProperties)))
{
}

Note: This object implements the IDisposable interface to ensure the life-cycle is managed properly.

In the above examples, it's understood that the configuration object specifies which client to construct. The ClientId property contained within the ClientConfiguration object provides MEF enough information to construct the appropriate client.

In some circumstances, you might not want to choose the client; rather, you would like TAPI to make the choice on your behalf. In this model, client metadata is retrieved via MEF and they're sorted from best to worst. Given a list of potential clients, TAPI runs a support check (see below) to determine if the client is fully supported in both Relativity and the client environment. Since this can take 10-15 seconds, the operation is asynchronous.

using (ITransferClient client = await host.CreateClientAsync(new ClientConfiguration(existingProperties)))
{
}

Note: This object implements the IDisposable interface to ensure the life-cycle is managed properly.

When using the dynamic transfer client, a strategy design pattern is used to determine the order in which clients are checked for compatibility. Out of the box, the default strategy is as follows:

• File share
• Aspera

The signature for this interface is as follows:

interface ITransferClientStrategy
{
    IEnumerable<ITransferClientMetadata> GetSortedCandidates(RelativityConnectionInfo connectionInfo, IEnumerable<ITransferClientMetadata> availableClients);
}

The caller is provided with the Relativity instance and all discovered transfer client plugin metadata. The returned collection is then used to provide a sorted list of candidates to use.

Given a transfer client object, the SupportCheckAsync method is called to perform the following checks:

• Verify the client is supported by Relativity.
• Verify the client is supported within the client-side environment.

The following example verifies the file share associated with the configured workspace is defined and accessible.

using (ITransferClient client = host.CreateClient(new FileShareClientConfiguration()))
{
    ISupportCheckResult result = await client.SupportCheckAsync();
    Console.WriteLine($"Support result: {result.IsSupported}");
}

A connection check performs a small upload and download to not only verify connectivity but to ensure support for reading from and writing to the storage. Likewise, a context object is provided to retrieve status messages. The check can be done on a per-client basis or a single method to exercise connection checks for all available clients.

The following example verifies the file share associated with the configured workspace is defined and accessible.

using (IRelativityTransferHost host = new RelativityTransferHost(connectionInfo))
using (ITransferClient client = host.CreateClient(new FileShareClientConfiguration()))
{
    DiagnosticsContext context = new DiagnosticsContext();
    context.DiagnosticMessage += (sender, args) => { Console.WriteLine($"Connection check event. Client: {args.ClientId}, Message: {args.Message}"); };

    // Perform a connection check only for the file share client.
    IConnectionCheckResult result = await client.ConnectionCheckAsync(context);
    Console.WriteLine($"Connection check result: {result.IsSuccessful}");

    // Perform a connection check for all available clients.
    foreach (IConnectionCheckResult result in await host.ConnectionChecksAsync(context))
    {
        Console.WriteLine($"Connection check result: {result.IsSuccessful}")
    }
}

The ITransferRequest object exposes optional SourcePathResolver and TargetPathResolver properties to take a given path and adapt it. The signature for this interface is as follows:

interface IRemotePathResolver
{
    string ResolvePath(string path);
}

The primary use-case for path resolvers is adding backwards compatibility to a client. For example, the Aspera client employs resolvers to adapt UNC paths to native Aspera UNIX-style paths.

*Note: The library provides AsperaUncPathResolver for API users that wish to use Aspera clients when using UNC paths exclusively. This is automatically used by Aspera jobs as long as ITransferRequest.RemotePathsInUncFormat is true.

The ITransferRequest object exposes an optional RetryStrategy property to define a method that dictates how much time is spent in between retry attempts. The signature for this interface is as follows:

interface IRetryStrategy
{
    Func<int, TimeSpan> Calculation { get; }
}

The TAPI library provides a RetryStrategies static class to provide two common strategies:

• Exponential backoff - This is the default strategy where each attempt waits longer by a power of 2.
• Fixed Time - This simply defines a constant wait period regardless of the number of attempts.

This object encapsulates a local or remote path and can be found throughout the API. This is a convenient extensibility point to support different kinds of paths.

Note: Both relative paths and paths transformed via IRemotePathResolver are automatically assigned to the TransferPath object.

There are several options to submit transfers and all revolve around the ITransferRequest object.

Although this object can be constructed directly, a number of static methods have been added to simplify construction using several overloads including:

• TransferRequest.ForDownload
• TransferRequest.ForDownloadJob
• TransferRequest.ForUpload
• TransferRequest.ForUploadJob

Once the transfer is complete, the ITransferResult object is returned and provides an overall summary.

This enumeration provides hints to better configure or optimize the transfer request.

This enumeration provides the overall transfer status.

Once a client is constructed, transfer jobs are created in one of two ways (see below). Once constructed, the job adheres to a common structure regardless of client. The job has several key responsibilities including:

• Provide a life-cycle for the entire transfer.
• Add paths to a job queue.
• Create local or remote path enumerators to search for paths or partition large datasets into smaller chunks.
• Change the data rate at runtime.
• Wait for all transfers to complete.
• Status and current statistics.
• Retry count.
• Retrieving a list of all processed job paths.

Note: This object implements the IDisposable interface to ensure the life-cycle is managed properly. If any operations are performed on the job after the object has been disposed, an ObjectDisposedException will be thrown.

If the list of source paths is already known, you simply construct the request object, call TransferAsync, and await the result.

using (ITransferClient client = host.CreateClient(configuration))
{
    const string SourcePath = @"C:\temp\file.txt";
    const string TargetPath = @"\\files\T003";
    TransferRequest request = TransferRequest.ForUpload(SourcePath, TargetPath);
    ITransferResult result = await client.TransferAsync(request);
    Console.WriteLine($"Status: {result.Status}");
}

This is an ideal approach to take if the number of files is small and the required functionality is minimal.

If the list of source paths is unknown, you want to avoid calling TransferAsync one file at a time. High-speed clients like Aspera require a significant amount of overhead to setup the transfer request and performance would suffer significantly. To address this scenario, the ITransferJob object can be constructed via the client instance. The general idea is to construct a job, continually add transfer paths to the queue as they become known, and then await completion. Beyond large transfers, the ITransferJob object provides several other functional advantages such as changing the data rate at runtime and a predictable object life-cycle.

It's understood that files are transferred as soon as they're added to the job queue.

// dr = datareader
TransferRequest request = TransferRequest.ForUploadJob(TargetPath);
using (ITransferClient client = host.CreateClient(clientConfiguration))
using (ITransferJob job = await client.CreateJobAsync(request))
{
    // Paths are added to the job as they become known.
    while (dr.Read())
    {
        string file = dr.GetString(0);
        job.AddPath(new TransferPath { Direction = TransferDirection.Upload, SourcePath = file, TargetFileName = Guid.NewGuid().ToString() });
    }

    // Wait until all transfers have completed.
    ITransferResult result = await job.CompleteAsync();
    Console.WriteLine($"Status: {result.Status}");
}

Note: Multiple jobs can be created; however, bandwidth constraints can lead to transfer errors.

The vast majority of workflows within Relativity center on the workspace. Additionally, the workspace configuration is responsible for defining several resource servers including the default file repository. API users can retrieve a Workspace object through IRelativityTransferHost to assist configuring the transfer request.

using (IRelativityTransferHost host = new RelativityTransferHost(this.connectionInfo))
{
    // Use the workspace specified within RelativityConnectionInfo.
    Workspace workspace1 = await host.GetWorkspaceAsync(token).ConfigureAwait(false);
    RelativityFileShare fileShare1 = workspace1.DefaultFileShare;

    // Or you can specify the workspace instead.
    const int WorkspaceId = 1234567;
    Workspace workspace2 = await host.GetWorkspaceAsync(WorkspaceId, token).ConfigureAwait(false);
    RelativityFileShare fileShare2 = workspace2.DefaultFileShare;
}

The Workspace object provides the following properties:

The Workspace object also defines the AdminWorkspaceId constant and the AdminWorkspace static property for situations where a workspace artifact cannot be specified or the API user would like to target a file share instead of using the default file share.

// Obtain the admin workspace artifact.
const int WorkspaceId = Workspace.AdminWorkspaceId;

// Or obtain the admin workspace object instead.
Workspace workspace = Workspace.AdminWorkspace;

When the IRelativityTransferHost object is constructed, the API user is required to supply a RelativityConnectionInfo instance. This object includes the workspace artifact identifier and is used by transfer clients to setup the transfer using the default file repository. This is known as auto-configuration and is ideal for API users because it simplifies the transfer setup process.

Although many Relativity applications are geared around the workspace, there are situations where the workspace may not be known, doesn't exist, or the application simply wants to specify a target file share. For more advanced transfer scenarios, the IFileStorageSearch API is used to retrieve file shares and the ClientConfiguration object is assigned the target file share.

The workspace represents both a functional and security boundary required by many Relativity API's. The resource pool is the set of servers and file repositories associated with the workspace. Bypassing workspaces and querying these objects directly represents a security risk and, as such, requires View Admin Repository Admin Operations rights. In short, the API user must have Admin Rights to search for all file shares or specify non-workspace search criteria.

This object represents a standard file share resource server and includes several RelativityOne cloud-specific properties.

The IFileStorageSearch API is constructed from the IRelativityTransferHost object and used to search for and identify file shares. Similar to other TAPI methods, a context object provides additional configuration details. Since none of the context properties are assigned, the example below searches for all file shares.

// In this example, the workspace is NOT supplied to RelativityConnectionInfo.
RelativityConnectionInfo connectionInfo = new RelativityConnectionInfo(host, credential);
using (IRelativityTransferHost host = new RelativityTransferHost(connectionInfo))
{
    IFileStorageSearch fileStorageSearch = host.CreateFileStorageSearch();
    FileStorageSearchContext context = new FileStorageSearchContext();
    FileStorageSearchResults results = await fileStorageSearch.SearchAsync(context, token).ConfigureAwait(false);
    foreach (RelativityFileShare fileShare in results.FileShares)
    {
        Console.WriteLine("File share name: " + fileShare.Name);
        Console.WriteLine("File share path: " + fileShare.Url);
    }
}

The search criteria is exposed through the FileStorageSearchContext object. Full system admin rights are required for all non-workspace search criteria and includes the following search options:

Note: All condition search options support standard Relativity query operators.

The following example uses the context to search for all files shares within the specified workspace:

// This does NOT require admin rights.
FileStorageSearchContext context = new FileStorageSearchContext { WorkspaceId = 1123815 };
FileStorageSearchResults results = await fileStorageSearch.SearchAsync(context, token).ConfigureAwait(false);

The following example uses the context to search for all files shares within any resource pool that matches the specified condition.

// This DOES require admin rights.
FileStorageSearchContext context = new FileStorageSearchContext { ResourcePoolCondition = "'Name' == 'default'" };
FileStorageSearchResults results = await fileStorageSearch.SearchAsync(context, token).ConfigureAwait(false);

The following example uses the context to search for the the file share that matches the specified resource server.

// This DOES require admin rights.
FileStorageSearchContext context = new FileStorageSearchContext { ResourceServerId = 1049366 };
FileStorageSearchResults results = await fileStorageSearch.SearchAsync(context, token).ConfigureAwait(false);

The following example uses the context to search for all files shares within the instance:

// This DOES require admin rights.
FileStorageSearchContext context = new FileStorageSearchContext { WorkspaceId = Workspace.AdminWorkspaceId };
FileStorageSearchResults results = await fileStorageSearch.SearchAsync(context, token).ConfigureAwait(false);

The SearchAsync returns a FileStorageSearchResults object to provide all search results.

This object provides helper methods to search for file shares by artifact or UNC path. The following example demonstrates retrieving the file share by the specified artifact identifier:

FileStorageSearchResults results = await fileStorageSearch.SearchAsync(context, token).ConfigureAwait(false);
const int FileShareArtifactID = 1049366;
RelativityFileShare fileShare = results.GetRelativityFileShare(FileShareArtifactID);

Given a file share object, the target file share is assigned within the ClientConfiguration object.

RelativityFileShare fileShare = results.FileShares.FirstOrDefault();
ClientConfiguration configuration = new ClientConfiguration();
configuration.TargetFileShare = fileShare;

Note: When specifying a target file share and using remote UNC paths, it's imperative for the UNC paths to share the same base path as the file share. This can be problematic for transfer clients like Aspera that doesn't support native UNC paths.

So far, the examples have included a small number of transfer paths. What if you want to create transfer paths from one or more search paths? More importantly, what about datasets consisting of hundreds of thousands or even millions of paths? Due to the RAM and CPU required to manage such large datasets, a different approach is required.

TAPI provides an enumeration feature, which delivers the following capabilities:

• Time-efficient enumeration of local paths,
• Filtering paths,
• Grouping paths into batches and serializing them,
• Current statistics of enumeration's progress.

Local enumeration is provided out of the box whereas remote enumeration should be provided by a customer of a given API.

The enumeration API is built upon fluent API pattern and can be easily customized. The EnumerationBuilder class serves the method to customize the enumeration for your needs:

The below samples depict the usage of EnumerationBuilder class, which exposes a fluent interface in order to build enumeration object.

The method ForUpload() will use the built-in, parallel enumerator to retrieve all paths for upload. It can be CPU-intensive, but it's the fastest method for it:

/// <summary>
/// Creates enumeration builder for local enumeration
/// </summary>
/// <param name="logger">Logger that will be used to log enumeration execution</param>
/// <param name="correlationId">CorrelationId used to associate events happening throughout the operation</param>
public static EnumerationBuilder ForUpload(ITransferLog logger, Guid correlationId)
 
usage:
var enumeration = EnumerationBuilder
                    .ForUpload(logger, Guid.NewGuid())
                    .StartFrom(filesOrDirectories)
                    .Create();

You can provide your own enumerator too - in case of specific needs (e.g. you already have all paths to transfer and you don't want to search through catalogs):

/// <summary>
/// Creates enumeration builder for local enumeration using custom enumerator
/// </summary>
/// <param name="enumerator">Custom enumerator</param>
/// <param name="logger">Logger that will be used to log enumeration execution</param>
/// <param name="correlationId">CorrelationId used to associate events happening throughout the operation</param>
public static IEnumerationNecessaryActionsBuilder ForUpload(IEnumeratorProvider enumerator, ITransferLog logger, Guid correlationId)

usage:
IEnumerationProvider customEnumerator = /* ... */;
var enumeration = EnumerationBuilder
                    .ForUpload(customEnumerator, logger, Guid.NewGuid())
                    .StartFrom(filesOrDirectories)
                    .Create();

In case of download, you need to provide a custom enumerator of remote paths to the method ForDownload():

/// <summary>
/// Creates enumeration builder for remote enumeration
/// </summary>
/// <param name="enumerator">Custom enumerator</param>
/// <param name="logger">Logger that will be used to log enumeration execution</param>
/// <param name="correlationId">CorrelationId used to associate events happening throughout the operation</param>
public static EnumerationBuilder ForDownload(IEnumeratorProvider enumerator, ITransferLog logger, Guid correlationId)
 
usage:
var enumeration = EnumerationBuilder
                    .ForDownload(remoteEnumerator, logger, Guid.NewGuid())
                    .StartFrom(filesOrDirectories)
                    .Create();

You might notice the method StartFrom(IEnumerable<INode> sourceNodes); - it takes a list of files and folders to be iterated. Content of the folders will be iterated as well.

The bare minimum to define an enumerator is to call methods:

1. ForUpload() or ForDownload(),
2. StartFrom().
3. Create()

Below you'll find how to specify additional capabilities.

Use the method WithFilters() to enrich enumeration object with filtering functionality. Provide collection of filters to be applied against each path and a callback function which will be executed every time a given path meets at least one filter.

/// <summary>
/// Enriches enumeration process with filtering
/// </summary>
/// <param name="filters">Enumerable of custom filters</param>
/// <param name="skippedItemHandler">Callback handler executed every time when enumerated path meets at least one filter</param>
public EnumerationBuilder WithFilters(IEnumerable<INodeFilter> filters, IEnumerationHandler<EnumerationIssue> skippedItemHandler)
 
usage:
var filterHandler = new EnumerationHandler<EnumerationIssue>(item=>
                        {
                            Console.WriteLine($"{item.Path} {item.ErrorMessage}");
                        });
var enumeration = EnumerationBuilder
                    .ForDownload(remoteEnumerator, logger, Guid.NewGuid())
                    .StartFrom(filesOrDirectories)
                    .WithFilters(filters, filterHandler)
                    .Create();

TAPI provides the following implementations of INodeFilter - you can define your own.

• AspxExtensionFilter - filters out all files with .aspx extension (to prevent errors of Aspera transfers),
• NoReadAccessFileNodeFilter - checks user's permission against a file (can slow down enumeration significantly),
• PathLengthFilter - finds paths longer than Aspera and file share support,
• R1PathSizeFilter - finds paths longer than RelativityOne supports.

You can define your handler to react on enumeration updates:

/// <summary>
/// Enriches enumeration process with statistics reporting
/// </summary>
/// <param name="statisticsHandler">Callback handler executed on every enumerated path to report enumeration overall statistic</param>
public EnumerationBuilder WithStatistics(IEnumerationHandler<EnumerationStatistic> statisticsHandler)

usage:
var statisticsHandler= new EnumerationHandler<EnumerationStatistic>(stats =>
            {
                Console.WriteLine($"Total bytes: {stats.TotalBytes} Total files: {stats.TotalFiles} Total Empty directories: {stats.TotalEmptyDirectories}");
            });
 
var enumeration = EnumerationBuilder
                    .ForUpload(logger, Guid.NewGuid())
                    .StartFrom(filesOrDirectories)
                    .WithStatistics(progressHandler)
                    .Create();

Grouping paths in batches is recommended for transfers with more than 1M files. Batches are stored on disk and you have to specify a handler to intercept all stored batches.

/// <summary>
/// Enriches enumeration process with dividing enumerated paths into chunks and converting them into batches
/// </summary>
/// <param name="destinationNode">The destination node.</param>
/// <param name="batchSerializationDirectory">A place to serialize series of paths that belong to batches</param>
/// <param name="batchCreatedHandler">Callback handler executed on every batch created</param>
/// <returns>This builder</returns>
public EnumerationBuilder WithBatching(INode destinationNode, IDirectory batchSerializationDirectory, IEnumerationHandler<SerializedBatch> batchCreatedHandler)
 
usage:
var batchCreatedHandler = new EnumerationHandler<SerializedBatch>(batch =>
    {
        Console.WriteLine($"{batch.BatchNumber} {batch.File} {batch.TotalFileCount}");
    });
 
var enumeration = EnumerationBuilder
                    .ForUpload(logger, Guid.NewGuid())
                    .StartFrom(filesOrDirectories)
                    .WithBatching(targetDirectoryOrDrive, directoryToStoreBatches, batchCreatedHandler)
                    .Create();

You can define batching parameters (i.e. max number of files and bytes in a single batch) within GlobalSettings class:

1. GlobalSettings.MaxFilesPerBatch
2. GlobalSettings.MaxBytesPerBatch

public interface EnumerationBuilder : IEnumerationNecessaryActionsBuilder, IEnumerationFinalActionsBuilder
{
    static IEnumerationNecessaryActionsBuilder ForUpload(ITransferLog logger, Guid correlationId);

    static IEnumerationNecessaryActionsBuilder ForUpload(IEnumeratorProvider enumerator, ITransferLog logger, Guid correlationId);

    static IEnumerationNecessaryActionsBuilder ForDownload(IEnumeratorProvider enumerator, ITransferLog logger, Guid correlationId);

	IEnumerationFinalActionsBuilder StartFrom(IEnumerable<INode> sourceNodes);

    IEnumerationFinalActionsBuilder WithFilters(IEnumerable<INodeFilter> filters, IEnumerationHandler<EnumerationIssue> skippedItemHandler);

    IEnumerationFinalActionsBuilder WithStatistics(IEnumerationHandler<EnumerationStatistic> statisticsHandler);

    IEnumerationFinalActionsBuilder WithBatching(INode destinationNode, IDirectory batchSerializationDirectory, IEnumerationHandler<SerializedBatch> batchCreatedHandler);

    IEnumerationOrchestrator Create();
}

The ClientConfiguration object supports setting a minimum and target data rate. There are situations where the API user would like to change the data rate at runtime. To facilitate this feature, the ITransferJob object allows each TAPI client to provide an implementation.

This feature is currently limited to the Aspera client and any attempt to call this method on a job/client that doesn't support setting or changing the data rate will throw NotSupportedException. To provide the API user a cleaner way to use this feature, the IsDataRateChangedSupported property is provided.

using (ITransferJob job = await client.CreateJobAsync(request))
{
    if (job.IsDataRateChangeSupported)
    {
        job.ChangeDataRate(0, 200);
        Console.WriteLine($"Changed the data rate. Min=0 Mbps, Target=200 Mbps.");
    }
}

All transfer events are exposed through the optional TransferContext object and include:

Note: The ITransferRequest object can take an optional TransferContext instance when event handling is required.

var context = new TransferContext();
context.LargeFileProgress += (sender, args) => { Console.WriteLine($"Large file progress event. Transfer path: {args.Path}, Chunk number: {args.ChunkNumber}, Total chunks: {args.TotalChunks}"); };
context.TransferPathProgress += (sender, args) => { Console.WriteLine($"Transfer path progress event. Transfer path: {args.Path}, Started: {args.StartTime}, Ended: {args.EndTime}"); };
context.TransferPathIssue += (sender, args) => { Console.WriteLine($"Transfer failure. Transfer path: {args.Issue.Path} Message: {args.Issue.Message}"); };
context.TransferJobRetry += (sender, args) => { Console.WriteLine("Transfer job retry. Count: {args.Count}"); };
context.TransferRequest += (sender, args) => { Console.WriteLine("Transfer request event. Status: {args.Status}"); };
context.TransferStatistics += (sender, args) => { Console.WriteLine($"Transfer statistics: {args.Statistics.Progress}"); };

// Pass the context when constructing a new request.
TransferRequest request = TransferRequest.ForUpload(SourcePath, TargetPath, context);

The TransferStatistics event is notable because it provides a wealth of useful runtime transfer info. The TransferStatisticsEventArgs class exposes an ITransferStatistics object and provides the following properties.

Notes:

• Not all transfer clients are guaranteed to supply all listed statistics.
• The rate at which statistics are raised is defined via StatisticsRateSeconds property defined on TransferContext.

The section above describes several statistics that applications using TAPI can use for common use-cases like driving the application front-end. However, any application that uses TAPI is equally interested in not only accessing statistics but other key transfer details to monitor the overall application health.

TAPI-enabled applications are able to take advantage of the APM framework by enabling the following opt-in transfer request setting:

ITransferRequest request;

// Application metrics can be identified using this property.
request.Application = "my application";

// Enable submitting APM metrics when the request has completed.
request.SubmitApmMetrics = true;

The following table outlines all TAPI APM metric fields:

Notes:

• New Relic APM support is provided for all Relativity One environments.

If a fatal exception occurs, such as OutOfMemoryException, ThreadAbortException, or a general connection exception, the rule is simple: TAPI throws TransferException. This must be handled by the API user.

When dealing with requests containing thousands or even millions of files, it is not enough to simply throw an exception when a file transfer error occurs. To support this type of error handling model, the ITransferIssue object encapsulates warning/error information. The API user can then decide how to handle these issues. It's understood that all issues are logged automatically.

The ITransferIssue object includes the following properties:

The IssueAttributes enumeration provides common issues that can be combined (for example, FlagsAttribute) and occur across any transfer client.

The following example demonstrates how issues are enumerated once the result is returned to the API caller.

TransferRequest request = TransferRequest.ForUpload(SourcePath, TargetPath);
ITransferResult result = await client.TransferAsync(request).ConfigureAwait(false);
foreach (ITransferIssue issue in result.Issues)
{
    if (issue.Attributes.HasFlag(IssueAttributes.Error))
    {
        Console.WriteLine($"File error: {issue.Path.SourcePath} - Message: {issue.Message}");
    }
    else if (issue.Attributes.HasFlag(IssueAttributes.Warning))
    {
        Console.WriteLine($"File warning: {issue.Path.SourcePath} - Message: {issue.Message}");
    }
}

If TAPI determines that a transfer error can be retried, it will attempt to retry the transfer for the configured number of attempts before throwing a TransferException. The following transfer errors can be configured to be retried:

The ITransferStatistics object has properties that indicate how many times the above errors have been encountered and retried. This object has a count of TotalBadPathErrors and TotalFilePermissionsErrors, which will be incremented every time these errors are encountered.

A number of common but optional settings are exposed by the GlobalSettings singleton.

Note: API users are strongly encouraged to set ApplicationName because the value is included within all log entries.

TAPI supports Relativity Logging and, specifically, the ILog object. There may be scenarios, however, where 3rd party developers may wish to use their own logging framework. The ITransferLog interface is an extensibility point to address this possible use-case. Similar to other TAPI objects, this interface implements IDisposable to manage object life-cycles.

Relativity Logging is the default logging implementation if none is explicitly provided. The next example demonstrates how a Relativity Logging ILog object is specified when creating the RelativityTransferHost object.

ILog log; // Retrieved or constructed via Relativity Logging.
using (ITransferLog log = new RelativityTransferLog(log))
using (IRelativityTransferHost host = new RelativityTransferHost(connectionInfo, log))
{}

Notes:

• If an ITransferLog object isn't provided, Relativity.Logging.Log.Logger is used to retrieve the current Relativity Logging instance.
• If a serious error occurs attempting to retrieve the ILog or an exception is thrown performing Relativity Logging setup, the NullTransferLog is constructed to avoid unnecessary fatal errors.

When a transfer request is made, all log entries are prefixed with useful property values to improve log searching and filtering to a particular request. This includes the following values:

• ApplicationName (via GlobalSettings)
• ClientRequestId (via ITransferRequest)
• JobId (via ITransferRequest)
• Direction (via ITransferRequest)

Note: The ApplicationName should be set to an appropriate value. The value defaults to TAPI if not specified.

Relativity Logging uses Serilog to format each log entry. Because Relativity Logging is the presumed default, the Serilog DSL is used throughout TAPI. When using a custom ITransferLog implementation, the message template and properties must be converted to a string; otherwise, an exception is thrown when calling the String.Format method. This StackOverflow page provides an example on how to use the MessageTemplateParser to convert the Serilog message and properties to a properly formatted string.

TAPI relies heavily on Relativity REST endpoints to support many of the standard features provided by the library. For RelativityOne applications, back-end services are continually evolving and increase the likelihood of compatibility issues. Although TAPI doesn't perform an automatic or implicit Relativity version check, a check is available and should be called by new applications as early as possible.

If the specified Relativity version is not supported, the method throws RelativityNotSupportedException and should be handled accordingly.

using (IRelativityTransferHost host = new RelativityTransferHost(connectionInfo))
{
    try
    {
        await host.VersionCheckAsync(token).ConfigureAwait(false);
    }
    catch (RelativityNotSupportedException)
    {
        // Handle the exception.
    }
}

Alternatively, a static method exists within the RelativityTransferHost class to perform the same test without requiring an IRelativityTransferHost instance.

try
{
    await RelativityTransferHost.VersionCheckAsync(connectionInfo, token).ConfigureAwait(false);
}
catch (RelativityNotSupportedException)
{
    // Handle the exception.
}

All DateTime objects values used by TAPI are in local time.

Relativity and the APIs consumed by Relativity use several different versions of the Newtonsoft.Json library, which can cause assembly version conflict at both build and run time. This is the recommended assembly binding redirect to Newtonsoft.Json version 6.0.0.0 to add to the consuming application (app.config) or web (web.config) configuration file. It ensures that almost any version of Newtonsoft.Json will work with the TAPI:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
      <startup>
      <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.2" />
      </startup>  
      <runtime>
            <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">             
                 <dependentAssembly>
                 <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
                       <bindingRedirect oldVersion="0.0.0.0-10.0.0.0" newVersion="6.0.0.0" />
                 </dependentAssembly>
           </assemblyBinding>
      </runtime>
</configuration>

There may be scenarios where the cost of transferring massive datasets consisting of small files is significantly higher than packaging them locally, transferring, and potentially extracting them on the server (extracting may not be necessary, depending on workflow). There may also be scenarios where the API user would like to alter file metadata, such as filename or file timestamps, before adding the native files to the package container.

TAPI extends the existing plugin-based design with a simple means to create or use package libraries through a familiar and consistent object model. Some of the key features include:

• MEF-based plugin design.
• Package dependencies are only required if packaging is used.
• Package files are transferred as soon as each package file is completed.
• Package events are exposed through a new package context object.
• Limited to uploads only.

The TAPI repository includes two NuGet packages to provide RCC package library functionality:

• relativity.transfer.package.rcc32 (32-bit package library)
• relativity.transfer.package.rcc64 (64-bit package library)

Similar to the ClientConfiguration object, the PackageConfiguration object is used to setup the package request and includes:

Similar to the TransferContext object, all package events are exposed through the optional PackageContext object and includes:

For those that may not be familiar with the RCC format, this was developed in the summer of 2015 to secure custodian-based collections. In a nutshell, it's similar to multi-file ZIP but offers a number of advantages including:

• The format is based on the SQLCipher port of SQLite.
• Cross platform support (Windows and OSX).
• Dependent on 32-bit and 64-bit native libraries.
• The executing application MUST assign the x86/x64 platform target (IE project settings)
• Full database AES-256 encryption in CBC mode via OpenSSL.
• You MUST specify a 16-character password.
• Custom metadata can be stored within the package.
• Extraction is done through a WPF-based application or RCC extraction API.
• Compression not yet supported.

A standard RCC is composed of three file types where each file type is uniquely identified. The SQLCipher database format provides a feature where the first 16 Bytes can be specified.

There are very few differences between a standard transfer request and one involving packaging. Ignoring package configuration and events, the main difference is calling PackageTransferAsync instead of TransferAsync.

var packageConfiguration = new PackageConfiguration
    {
        Compression = false,
        DeletePackageFiles = true,
        FileName = "Sample.rcc",
        Name = "Sample Dataset",
        PackageLibrary = WellKnownPackageLibrary.Rcc,
        Password = "P@ssw0rd@1012345"
    };

    // Custom metadata can be added to the package.
    var packageMetadata = new Dictionary<string, object>();

    // The package context exposes events, similar to how the context is used throughout TAPI.
    var context = new PackageContext(packageConfiguration, packageMetadata);

    // This event is raised when a new package is created.
    context.PackageCreated += (sender, e) =>
        {
            Console.WriteLine($"Package {e.File} has been created.");
        };

    // This event is raised when package progress occurs.
    context.PackageProgress += (sender, e) =>
        {
            Console.WriteLine("Package progress: {0:#.##}%", e.Progress);
        };

    // This event is raised as package files are about to be get added to the package.
    context.PackagingFile += (sender, e) =>
        {
            e.SourceFileMetadata.CreationTime = DateTime.Today;
            e.SourceFileMetadata.LastAccessTime = DateTime.Today.Subtract(TimeSpan.FromDays(1));
            e.SourceFileMetadata.LastWriteTime = DateTime.Today.Subtract(TimeSpan.FromDays(2));
            e.SourceFileMetadata.FileName = "modified-" + e.SourceFileMetadata.FileName;
            e.SourceFileMetadata.ExtractDirectory = null;
            Console.WriteLine($"Adding source file '{e.SourceFileMetadata.File}' to package.");
        };

    // Simultaneously package the request and transfer all package files.
    var request = TransferRequest.ForUpload(
        new TransferPath { SourcePath = @"C:\MyDataset" },
        @"\\files\PackageUpload");
    var result = await client.PackageTransferAsync(request, context);