Setting up the Client
You can download the Client using docker. To download the client, use from the command line:
docker pull quay.io/ucsc_cgl/core-client:1.1.2
Getting the token
Please login using your UCSC Google account (your @ucsc.edu address) and download your Redwood token via the "GET TOKEN" link on the top right corner of the page. If you see "No Token" after login please contact Brian O'Connor from your UCSC email account and request one.
IMPORTANT: Treat your token as you would a credit card number. Do not share or send your token around including electronic means such as e-mail, GitHub, etc. In case your token is compromised, please contact Brian O'Connor as soon as possible.
Perform Bulk Download via Manifest
This assumes the current working directory (pwd) has a manifest, like the one you can download from the file browser. The command below will download the files to the current working directory under the subdirectory called outputs.
NOTE: make sure you have enough space in pwd!
docker run --rm -it -e ACCESS_TOKEN=<access_token> -e REDWOOD_ENDPOINT=ucsc-cgp.org -v $(pwd)/<my_manifest_file.tsv>:/dcc/dcc-spinnaker-client/data/manifest.tsv -v $(pwd)/samples:/samples -v $(pwd)/outputs:/outputs -v $(pwd):/dcc/data quay.io/ucsc_cgl/core-client:1.1.2 redwood-download /dcc/dcc-spinnaker-client/data/manifest.tsv /outputs
Upload via Spinnaker
Create a manifest that links your metdata and data. Your manifest.tsv should be a TSV based on this template. You can hover over the column names to see more information. Here are more details about the columns that use a controlled vocabulary:
You need to include file paths to your upload files that start with /samples since that's the location used in the docker run below. Columns Donor UUID, Specimen UUID, Sample UUID, Upload File ID, Data Bundle ID, and Metadata.json should be left blank! Workflow Version should be 1.0.0
You should create a sub-directory called samples under your current working directory (pwd) with your files to be uploaded. Your manifest.tsv should be in your current working directory (pwd). Then execute the following:
docker run --rm -it -e ACCESS_TOKEN=<access_token> -e REDWOOD_ENDPOINT=ucsc-cgp.org -v $(pwd)/<my_manifest_file.tsv>:/dcc/manifest.tsv -v $(pwd)/samples:/samples -v $(pwd)/outputs:/outputs quay.io/ucsc_cgl/core-client:1.1.2 spinnaker-upload --force-upload --skip-submit /dcc/manifest.tsv
Once completed, you will find a receipt file ($(pwd)/outputs/receipt.tsv) which you should save. It provides various IDs assigned to your donor, specimen, sample and file that make it much easier to find/audit later.
In case your upload failed and need to re-upload your files, make sure you edit the 'Workflow Version' entry by increasing it by 0.0.1. For example, if you previously uploaded a file with workflow version 1.0.1, you would modify the entry in the TSV file so the workflow version is 1.0.2 instead. In addition, make sure you save the generated receipt somewhere safe for your records, and delete the generated 'outputs' folder.
NOTE: You may use dstat to monitor network usage to ensure uploads are in progress.
You can get the latest validated Metadata dump by clicking here.
You can get the latest file browser index dump by clicking here.