Home Browser Boardwalk Redwood Spinnaker Wiki About Help Login


Setting up the Client

You can download the Client using docker. To download the client, use from the command line:

docker pull quay.io/ucsc_cgl/core-client:1.1.2

Getting the token

Please login using your UCSC Google account (your @ucsc.edu address) and download your Redwood token via the "GET TOKEN" link on the top right corner of the page. If you see "No Token" after login please contact Brian O'Connor from your UCSC email account and request one.

IMPORTANT: Treat your token as you would a credit card number. Do not share or send your token around including electronic means such as e-mail, GitHub, etc. In case your token is compromised, please contact Brian O'Connor as soon as possible.

Perform Bulk Download via Manifest

This assumes the current working directory (pwd) has a manifest, like the one you can download from the file browser. The command below will download the files to the current working directory under the subdirectory called outputs.

NOTE: make sure you have enough space in pwd!

docker run --rm -it -e ACCESS_TOKEN=<access_token> -e REDWOOD_ENDPOINT=ucsc-cgp.org -v $(pwd)/<my_manifest_file.tsv>:/dcc/dcc-spinnaker-client/data/manifest.tsv -v $(pwd)/samples:/samples -v $(pwd)/outputs:/outputs -v $(pwd):/dcc/data quay.io/ucsc_cgl/core-client:1.1.2 redwood-download /dcc/dcc-spinnaker-client/data/manifest.tsv /outputs

Upload via Spinnaker

Create a manifest that links your metdata and data. Your manifest.tsv should be a TSV based on this template. You can hover over the column names to see more information. Here are more details about the columns that use a controlled vocabulary:

Help Information Submitter Donor Primary Site Primary site of the collected tissue associated with the file. Use the BRENDA tissue ontology. E.g. prostate would be officially "prostate gland" which corresponds to BTO:0001129. You can find it here. Submitter Specimen Type Describes the specimen type associated with the file. Use specimen.0.specimen_type.v3 ontology from here. Submitter Experimental Design This is the Experimental Design used on the particular file. Values should come from the GLOBAL.0.sequencing_strategy.v1 ontology referenced here. Analysis Type The analysis type you want to perform on the submitted file. It can be:

  • sequence_upload
  • alignment
File Type The file type that will be uploaded. Values should come from EDAM ontology.

You need to include file paths to your upload files that start with /samples since that's the location used in the docker run below. Columns Donor UUID, Specimen UUID, Sample UUID, Upload File ID, Data Bundle ID, and Metadata.json should be left blank! Workflow Version should be 1.0.0

You should create a sub-directory called samples under your current working directory (pwd) with your files to be uploaded. Your manifest.tsv should be in your current working directory (pwd). Then execute the following:

docker run --rm -it -e ACCESS_TOKEN=<access_token> -e REDWOOD_ENDPOINT=ucsc-cgp.org -v $(pwd)/<my_manifest_file.tsv>:/dcc/manifest.tsv -v $(pwd)/samples:/samples -v $(pwd)/outputs:/outputs quay.io/ucsc_cgl/core-client:1.1.2 spinnaker-upload --force-upload --skip-submit /dcc/manifest.tsv

Once completed, you will find a receipt file ($(pwd)/outputs/receipt.tsv) which you should save. It provides various IDs assigned to your donor, specimen, sample and file that make it much easier to find/audit later.

In case your upload failed and need to re-upload your files, make sure you edit the 'Workflow Version' entry by increasing it by 0.0.1. For example, if you previously uploaded a file with workflow version 1.0.1, you would modify the entry in the TSV file so the workflow version is 1.0.2 instead. In addition, make sure you save the generated receipt somewhere safe for your records, and delete the generated 'outputs' folder.

NOTE: You may use dstat to monitor network usage to ensure uploads are in progress.

Downloading Metadata

You can get the latest validated Metadata dump by clicking here.

You can get the latest file browser index dump by clicking here.