Upload via the command line

Overview

The Seven Bridges CLI Uploader is the recommended tool for uploading data to the Seven Bridges Platform. It is integrated into the Seven Bridges CLI.

The Command Line Uploader is recommended for large scale uploads. If you need to upload smaller scale uploads instead, we recommend using the Web uploader.

The maximum number of files you can submit for upload is 250,000.

COMMAND LINE OPTIONS

Syntax:

sb [global-parameters] upload <upload-subcommand> [command-parameters]

The supported global parameters are:

Option	Short parameter	Description
`--config <string>`		Configuration file to use instead of the default one.
`--help`	-h	Display help.
`--profile <string>`		Configuration profile to use from credentials file. Please note that this parameter is applicable only to "start" subcommand, "resume" and "delete" subcommands will use the same profile as the one used for starting the upload job, while for "status" subcommand this parameter is not applicable. (default "default")
`--debug`		Run the command with debug information in the output.

Subcommands

Subcommand	Description
`start`	Start the upload job
`status`	Check the upload status
`resume`	Resume the upload job
`delete`	Delete an upload job. Please note that only jobs that are paused can be deleted. To pause an upload job, use CTRL+C.

Read below for detailed information and instructions for all of the commands.

Start the upload

This command initializes a new upload job. Each job is identified by its unique name which can be assigned manually (using the corresponding command parameter) or automatically. Job name can be used to track the upload status and resume its execution (in case it is paused).

The upload is performed in the foreground, in the current active command shell session. The command shell session needs to remain open during the upload command execution, otherwise the upload will be paused and can be resumed. During the upload providing any other input will not be possible.

You can perform multiple uploads (i.e. start multiple upload jobs) on a single machine. The following are the statuses for an upload job:

Initializing -preparing the list of items (files and folders) which will be uploaded
Running- the items are being uploaded.
Paused - the upload job can be interrupted by terminating active command shell or issuing the CTRL+C command
Completed- this status denotes that all items are processed and the upload job is done; it doesn't necessarily mean all items have been uploaded successfully. The log file contains more detailed information about the upload job, including failed items. The status command provides an overview of the upload job, including the number of uploaded, failed, skipped, and remaining items.

sb [global-parameters] upload start [<input> | --manifest-file <manifest-file>] --destination <path> [command-parameters]

Required parameters

Name	Description
`--destination <string>`	Upload destination, which can be project root or a specific folder inside a Platform project. Destination can be specified either by name or ID (e.g. "sbguser/sbgproject" or "sbguser/sbgproject/directory1" or "5dc01c9ae4b01e2d090a700a").

Optional parameters

Name	Description
`<input>`	The Item (file or folder) or list of items to be uploaded. By default, the source folder structure is preserved on the Platform. The `<input>` and `--manifest-file` parameters are mutually exclusive, and one of them must be included within the `start` command.
`--manifest-file <manifest-file>`	The manifest file which contains the list of items that will be uploaded as well as the accompanying metadata (only applies to files). By default, the source folder structure is preserved on the Platform after the upload. The `<input>` and `--manifest-file` parameters are mutually exclusive, and one of them must be included within the `start` command. Learn about the manifest file format.
`--name <string>`	The upload job name which must be unique. The allowed characters are A-Za-z0-9_-. The maximum number of characters is 20. If omitted, the name will be automatically assigned using the following format "DAY_DDMMMYYYY_HHMMSS", for example Thu_27Feb2020_231455).
`--autorename`	If a file with the same name already exists at the upload destination, an underscore followed by a serial number will be added as a prefix. In case of a folder, the contents of both folders will be merged. In case a file and a folder bear the same name, the upload will be skipped. The `--autorename` and `--overwrite` parameters are mutually exclusive, if omitted SKIP will be used as default method.
`--overwrite`	If a file with the same name already exists at the upload destination, it will be overwritten by the file that is uploaded. In case of a folder, the contents of both folders will be merged. In case a file and a folder bear the same name, the upload will be skipped. The `--autorename` and `--overwrite` parameters are mutually exclusive, if omitted SKIP will be used as default method.
`--tag <string>`	Tag which will be set for each file once its upload is complete. Please note that tagging is not applicable to folders on the Seven Bridges Platform. It is possible to set maximum 32 tags, with each tag being maximum 64 characters long.
`--chunk-size <int>`	Preferred size of the upload part in bytes. If omitted, the default value of 64MiB will be used. Unstable network connection: use for example --chunk-size 8000000 (note that min value is 5243000) . If you have an unstable network connection, we recommend using '--chunk-size 8000000'.
`--parallel`	Maximum number of parallel file uploads. The allowed range for this value is 1 to 8. The default value is 8.  If you have a low upload speed or low (filesystem) read speed (eg. magnetic discs) or if you want to limit system resources used by the uploader, we recommend using `--parallel 4` or `--parallel 2`.
`--speed-limit <int>`	Maximum allowed network bandwidth for the process that executes this upload job. Should be specified in kbps (kilobits per second). If omitted, the maximum possible bandwidth will be used.

The following table illustrates the entire naming conflict resolution mechanism:

Check the upload status

Use the status command to check the upload status. You can check the status for all upload jobs by omitting the name parameter.

sb [global-parameters] upload status [<name>]

This will return the following information about each of the upload jobs:

Upload job name
Status
Processed (percent completed, bytes uploaded / total bytes)
Average upload speed (total bytes uploaded / total time spent in "Running" status)
ETA (Estimated Time of Arrival - expected remaining time for job completion)
To check a specific upload job, you should specify the name of the upload job. This will return more detailed information about the upload job:
Upload job name
Status
Log file path
Time submitted
Upload command (note: if wildcard is used in upload command, expanded command will be displayed)
Total number of submitted files
Number of uploaded files (successfully uploaded to the Platform)
Number of skipped files
Number of failed files
Number of remaining files (files in "Queued" and "Uploading" state)
Processed (percent completed, bytes uploaded / total bytes)
Average upload speed - total bytes uploaded / total time spent in "Running" status
ETA (Estimated Time of Arrival - expected remaining time for job completion)
The information about completed jobs will be kept in job history for at least 1 month after it has been completed or paused, unless it's been deleted (using the sb upload delete option, see below). After this time, the completed and paused jobs will be removed from the list of upload jobs.

Optional parameters

Name	Description
`<name>`	The name of the upload job. Use this parameter to obtain more detailed information about a specified job.

Resume an upload job

Use this command to resume a previously paused upload job. The upload will resume its execution from where it had been paused.

sb [global-parameters] upload resume [<name>]

Optional parameters

Name	Description
`<name>`	Specify the name of the paused job you wish to resume. This parameter is required unless there's only one paused job.

Delete an upload job

Use this command to delete a specified upload job or all upload jobs. Only jobs that have been completed or paused can be deleted. To pause the job, use CTRL+C.

sb [global-parameters] upload delete [<name> | --all ]

Please keep in mind that an upload job is kept in the job history for (at least) 1 month after it has been completed or paused and is automatically deleted from the list after that time.

Optional parameters

Name	Description
`<name>`	Upload job name, required if the -all parameter is not used. The `<name>` and `-all` parameters are mutually exclusive. One or the other is required.
`-all`	Deletes all upload jobs which are COMPLETED or PAUSED. The `<name>` and `-all` parameters are mutually exclusive. One or the other is required.

Examples

This section will list several examples on how to upload files. The destination project that is used in all of the examples is: rfanklin/my-project.

Please note that tags will not be shown in the examples.

Initial state

The following snippet shows the example of a directory tree on a local computer (from the local path where sb commands are executed):

├── dir1
├── dir2
│   ├── dir2-1
│   │   └── file4.bam
│   └── dir2-2
├── file1.bam
├── file2.bam
└── file3.bam

Example 1

Upload a folder and 2 files into the project root and tag the uploaded files.

Command

sb upload start dir1/ file1.bam file2.bam --destination rfranklin/my-project --tag upload1

Terminal output

upload job name: Thu_08Apr2021_182030
COMPLETE
Successfully uploaded 2 of 2 files

Result on the Platform

├── dir1
├── file1.bam
└── file2.bam

Example 2

Upload a folder (with underlying folder structure and one file included) and 3 files into the existing folder within the project.

assigning name to the upload job (for easier status tracking)
tagging uploaded files
and specifying a custom setting for the number of parallel file uploads (covering the
low upload speed case).

Command

sb upload start dir2 file1.bam file2.bam file3.bam --destination rfranklin/my-project/dir1 --name upload2 --tag upload2 --parallel 2

Terminal output

upload job name: upload2
COMPLETE
Successfully uploaded 4 of 4 files

Result on the Platform

The following will be the result on the Platform, assuming that the Example 1 (see above) is already executed and that the result remains intact:

├── dir1
│   ├── dir2
│   │   ├── dir2-1
│   │   │   └── file4.bam
│   │   └── dir2-2
│   ├── file1.bam
│   ├── file2.bam
│   └── file3.bam
├── file1.bam
└── file2.bam

Example 3

Upload 2 folders (with underlying folder structure and one file included) and 1 file into the project root, while:

choosing auto-rename as a method for resolving name conflict
tagging uploaded files
and specifying a custom setting for the chunk size (covering the unstable network connection case).

Command

sb upload start dir1/ dir2/ file1.bam --destination rfranklin/my-project --autorename --tag upload3 --chunk-size 8000000

Terminal output

upload job name: Thu_08Apr2021_182550
COMPLETE
Successfully uploaded 2 of 2 files

Result on the Platform

The following will be the result on the Platform, assuming that the two previous examples are already executed:

├── _1_file1.bam
├── dir1
│   ├── dir2
│   │   ├── dir2-1
│   │   │   └── file4.bam
│   │   └── dir2-2
│   ├── file1.bam
│   ├── file2.bam
│   └── file3.bam
├── dir2
│   ├── dir2-1
│   │   └── file4.bam
│   └── dir2-2
├── file1.bam
└── file2.bam

Example 4

Check the status for all upload jobs.

Command

sb upload status

Terminal output

Upload job name       Status                 Processed  Average speed  Estimated time  
                                                                                       
Thu_08Apr2021_182030  COMPLETED    100% (85.00/85.00B)      85.00 Bps  N/A  
upload2               COMPLETED  100% (114.00/114.00B)      38.00 Bps  N/A  
Thu_08Apr2021_182550  COMPLETED    100% (68.00/68.00B)      68.00 Bps  N/A

Example 5

Get the detailed status for an upload job.

Command

sb upload status upload2

Terminal output

Upload job name:       upload2  
Status:                COMPLETED  
Log file path:         /home/nikola/.sevenbridges/sb/logs/sb.log  
Time submitted:        08.04.2021 18:23  
Command:               sb upload start dir2 file1.bam file2.bam file3.bam --destination rfranklin/my-project/dir1 --name upload2 --tag upload2 --parallel 2  
Total files:           4  
Total size:            114.00 B  
# uploaded:            4  
# skipped:             0  
# failed:              0  
# remaining:           0  
% processed:           100%  
Average upload speed:  38.00 Bps  
ETA                    N/A

Example 6

Delete single upload job and check the status.

Commands

sb upload delete upload2
sb upload status

Terminal output

Upload job name       Status               Processed  Average speed  Estimated time  
                                                                                     
Thu_08Apr2021_182030  COMPLETED  100% (85.00/85.00B)      85.00 Bps  N/A  
Thu_08Apr2021_182550  COMPLETED  100% (68.00/68.00B)      68.00 Bps  N/A

Example 7

Delete all upload jobs and check the status.

Commands

sb upload delete --all
sb upload status

Terminal output

Upload job name       Status               Processed  Average speed  Estimated time

Updated over 3 years ago