README.md

Overview

AXL defines a common interface for transferring files between layers in a storage hierarchy. It abstracts vendor-specific APIs and provides synchronous and asynchronous methods using POSIX. One creates a transfer object, defining the transfer type, and then one adds files to the transfer. Once all files have been added, one initiates the transfer and can then later test or wait for its completion. The library optionally records the state of ongoing transfers, so that they can be identified or terminated even if the process that initiated the transfer has been restarted.

For an example of how to use AXL, see test_axl_sync.c

For more detailed use of the API, refer to axl.h.

Initializing AXL

Before calling AXL funcitons, one must first initialize the library by calling AXL_Init. One can optionally provide the path to a file where AXL can persist its state. If given a state file, AXL can recover its state upon restarting the process. If no state files is needed, one may pass NULL in place of the path name.

One must call AXL_Finalize to shut down the library.

Setting options

AXL contains several tunable options that control how files are transferred. These are settable using the AXL_Config function which takes as its single argument a pointer to a kvtree of option-value pairs. On success a non-NULL pointer is returned, on failure NULL is returned. Calling AXL_Config with a NULL pointer instead returns a kvtree with the current values of all settable configuration options. It must be freed used kvtree_delete after use. A subset of options can also be set for individual AXL transfers by passing them in a subtree "id" of the kvtree indexed by their integer transfer id. These options are included in the result of AXL_Config as an "id" subtree. By default a transfer will use the values of the global configuration options present at the time the transfer is created. Changing options of a transfer after is has been dispatched results in undefined behavior.

An example kvtree may look like this:

+- FILE_BUF_SIZE
|  +- 1000000
+- DEBUG
|  +- 0
+- id
   +- 42
   |  +- FILE_BUF_SIZE
   |  |  +- 1048576
   |  +- COPY_METADATA 
   |     +- 1
   +- 1
      +- FILE_BUF_SIZE
         +- 65536

axl.h defines symbolic names for all parameter which should be used instead of the immediate strings whenever possible to avoid typos and runtime errors.

The current set of configuariont options including their default value is (prefix Name by AXL_KEY_CONFIG_ for symbolic name):

Name	Type	Default	Per transfer	Description
FILE_BUF_SIZE	Byte count	1048576	Yes	Specify the number of bytes to use for internal buffers when copying files between source and destination.
DEBUG	Boolean	0	No	Set to 1 to have AXL print debug messages to stdout, set to 0 for no output.
MKDIR	Boolean	1	Yes	Specifies whether the destination file system supports the creation of directories (1) or not (0).
COPY_METADATA	Boolean	0	Yes	Whether file metadata like timestamp and permission bits should also be copied.

Thread safety: setting the DEBUG or any per-transfer configuration value after the transfer has been dispatched entails a race contion between the main thread and the worker threads. Changing configuration options after a transfer has been dispatched is not supported.

Transferring files

Regardless of the transfer type, the basic control flow of a transfer is always:

1. AXL_Create - allocate a new transfer object, providing its type and a name
2. AXL_Add - add a file to a transfer object, giving both source and destination path
3. AXL_Dispatch - start the transfer
4. AXL_Test - optional, non-blocking test for whether AXL_Wait will block
5. AXL_Cancel - optionally cancel a dispatched transfer
6. AXL_Wait - wait for transfer to complete
7. AXL_Free - free resources associated with transfer object allocated in AXL_Create

AXL_Create returns a transfer id that is the identifier for the transfer object for most other calls. It returns -1 if it fails to create a transfer object. The name is meant to serve as a user-friendly string.

One may add multiple files to a transfer, and a transfer having zero files is also valid.

One may cancel an outstanding transfer by calling AXL_Cancel between AXL_Dispatch and AXL_Wait. One must still call AXL_Wait on a cancelled transfer.

AXL_Test does not indicate whether a transfer succeeded. It indicates whether a call to AXL_Wait will block. One must call AXL_Wait to identify whether a transfer succeeded or failed.

It is valid to free a transfer object if it has not been dispatched.

One can call AXL_Stop to terminate any and all outstanding transfers without having to know the status, identifiers, or names of those transfers. One can not call wait or free on transfers that were terminated with AXL_Stop.

If a transfer fails, partially transferred files are not removed from the destination.

Transfer types

• AXL_XFER_SYNC - this is a synchronous transfer, which does not return until the files have been fully copied. It uses POSIX I/O to directly read/write files.

• AXL_XFER_PTHREAD - Like AXL_XFER_SYNC, but use multiple threads to do the copy.

• AXL_XFER_ASYNC_BBAPI - this method uses IBM's Burst Buffer API to transfer files. IBM's system software then takes over to move data in the background. It's actually using NVMeoF, reading data from the local SSD from a remote node, so that the compute node is not really bothered once started. If either the source or destination filesystems don't support the BBAPI transfers, AXL will fall back to using a AXL_XFER_PTHREAD transfer instead.

• AXL_XFER_ASYNC_DW - this method uses Cray's Datawarp API.

• AXL_XFER_DEFAULT - Let AXL choose the fastest transfer type that is compatible with all VeloC transfers. This may or may not be the node's native transfer library.

• AXL_XFER_NATIVE - Use the node's native transfer library (like IBM Burst Buffer or Cray DataWarp) for transfers. These native libraries may or may not support all VeloC transfers.