The data submission portal enables laboratories to submit data to the Brain Image Library. The key processes to submit data are:
The following terms in this document have specific context, as defined below:
Dataset
At BIL a dataset is a stand-alone entry of an image-volume or image-set associated with a single subject or experimental unit with unique metadata. A single dataset is usually associated with a single donor or subject when submitting multiple subjects in a collection, or a single part of the brain when imaging many parts of the brain in a collection. Many datasets make up a collection. A dataset typically contains many 2d image files that are assembled to form a more complex two or three dimensional volume.
Submission
A submission contains one or more related datasets and the associated metadata. Submissions will inherit project metadata (such as the NIH project, grant number, laboratory name, etc.), thus all datasets within a submission must belong to the same project. A submission can contain a single or multiple datasets. In general, smaller submissions are recommended because all datasets within a submission must pass the validation process for the datasets within the submission to be published. Each “level” of data should be uploaded in separate submissions (e.g. The set of raw data and the same data aligned to a reference are considered two separate submissions).
Project
A way of grouping data to ensure that (1) People working with the data get gain appropriate access and credit for their contributions and (2) Data is linked to the proper funding sources.
Publish
The act of making a submission that has passed all validation checks publicly available.
Once you have created your account by creating an ACCESS portal account, requested access to the BIL submission portal, and set your initial password, you are ready to continue to the submission portal.
Visit submit.brainimagelibrary.org and enter your PSC username and password.
Only users with PI/Data manager access with have the functions described in this step. If you need access to these functions,
have the PI send email to the bil-support@psc.edu with your BIL userid and name to request access.
The PI dashboard allows PIs and Data Managers to define and manage projects. This dashboard will allow the PI to add authorized users to the project and see the status of in-progress, unsuccessful, and successful submissions, regardless of the users that uploaded and submitted the data.
To enter (or edit) project information, Select the Manage Projects
option. From this menu you can create a new project and view both the personnel and submissions associated with each project.
To define a project select the Create a New Project
button. Enter the project name and the grant number associated with the project in the Funded By:
field.
If the project is a BICCN project, please be sure to tag the project appropriately and make sure that the Project Name
field contains the project name assigned by BCDC.
To add a user to the project (such as a data submitter) select the Manage Projects
link from the main dashboard then select View Personnel
for the project you would like to add the user to. Select the user from the list and select Submit
at the bottom of the page.
There MUST be a project defined to create a BIL submission. See step 2 above for more details.
A submission contains one or more related datasets and associated metadata. Submissions will inherit project metadata (such as grant), thus all datasets within a submission must belong to the same project.
A submission can contain a single or multiple datasets, but smaller submissions are recommended because all datasets within a submission must pass the validation process for the datasets within the submission to be published.
To create a submission, select the Submission sub-option of the New menu.
Next, enter the required metadata associated with the submission collection (see image below).
When you are done filling out the form, select the Save button. You will then be automatically taken to the metadata submission step.
When created, each BIL submission will have a a unique 16-digit identifier associated with it (e.g. 6247417d691a4548) and a unique dropbox-like landing zone directory (e.g. /bil/lz/username/6247417d691a4548. This landing zone directory is where the datasets belonging to the submission must be transferred to for validation and ingestion processing.
After the collection is created, you will automatically be taken to the metadata submission page. Alternativey, you can also access the metadata submission page from the submission portal by selecting New
and then New Metadata
in the drop-down menu.
If you are NOT ready to upload metadata at this time, click Cancel to exit the upload metadata spreadsheet step. When you are ready to load your metadata, return to the portal and select New
and then New Metadata
in the drop-down menu.
You can download the metadata sheet in the submit portal at this point. Next, you will select the appropriate metadata-schema for your data. Be sure to choose the appropriate option as this will affect the way your metadata in ingested.
Metadata Schema Options
Each dataset was generated from a single sample and donor (e.g. whole brain imaging) on the same instrument. Multiple datasets can be accommodated in a single submission. The lines of the specimen tab in the metadata spreadsheet must match exactly to the lines in the dataset tab (i.e. the specimen listed in specimen tab row 5 is the specimen for dataset row 5). Each dataset tab should have a corresponding image tab. There should be exactly one line in the instrument tab.
The dataset was generated from multiple samples, specimens, or donors on the same instrument. Only a single dataset can be accommodated per submission. Multiple specimens should be listed in the metadata spreadsheet, but only a single dataset. There should be one line in the image tab. There should be exactly one line in the instrument tab.
All datasets were generated from one specimen, but are from unique regions of interest on the specimen. Multiple datasets can be accommodated in a single submission. List the specimen multiple times in the specimen tab. Identify the specific region of interest in the locations column. Multiple donor lines must match exactly in the metadata spreadsheet (i.e. the specimen listed in specimen tab row 5 is the specimen for dataset row 5). Each dataset tab should have a corresponding image tab. There should be exactly one line in the instrument tab.
All datasets were generated from one specimen – but use different experimental (machinery) parameters. Multiple datasets can be accommodated as a single submission. There should be exactly one entry in the specimen tab. The lines listed in the Instrument tab in the metadata spreadsheet must match exactly to the lines in the dataset tab (i.e. the instrument listed in instrument tab row 5 is the instrument for dataset row 5). Each dataset tab should have a corresponding image tab.
After a metadata spreadsheet is prepared, select the submission collection it will be associated with. Then select the Upload Metadata
button to select your metadata spreadsheet and upload it. This button will not be available until a metadata schema is selected.
Note that while preparing the metadata spreadsheet, separate data directories need to be listed for each dataset in the submission.
The submission portal creates an upload landing zone space for you to transfer your image data to. To find this landing zone, select the Submissions sub-option of the View menu. The field “Data Path” shows the landing zone space for the submission.
The ingestion process currently supports native TIFF and JPEG 2000 image file formats and formats that are convertible to the OME-TIFF format via the bioformats tool. For more information on the bioformats tool see:
https://docs.openmicroscopy.org/bio-formats/5.7.3/formats/index.html.
Note that separate data subdirectories are required for each dataset in the submission. For example, if you had an experiment containing 5 mouse datasets that you wanted to to include as a single submission collection, you would create 5 subdirectories in the landing zone for the submission collection, one for each mouse dataset. e.g:
/bil/lz/testuser/abcdef0123456789/mouse1
/bil/lz/testuser/abcdef0123456789/mouse2
/bil/lz/testuser/abcdef0123456789/mouse3
/bil/lz/testuser/abcdef0123456789/mouse4
/bil/lz/testuser/abcdef0123456789/mouse5
The recommended practices to structure your data directories are further described here.
To tie metadata to an image dataset, each image dataset must be uploaded in a separate subdirectory.
Due to size, image data can not be uploaded through the submission portal (submit.brainimagelibrary.org). It must be uploaded separately through the BIL data transfer nodes, which are available at the virtual host: upload.brainimagelibrary.org. All users authorized to use the data submission portal (submit.brainimagelibrary.org) are also authorized to use the data transfer nodes (upload.brainimagelibrary.org). The username and passwords are the same on both systems.
There are many supported ways to upload files into the landing zone directory through the BIL data transfer nodes including rsync, Globus, sftp, and scp.
Using rsync to upload.brainimagelibrary.org: An example uploading all data in the directory (mouse1) as user testuser through the data transfer node upload.brainimagelibrary.org to the submission collection landing zone (abcdef0123456789)is shown below:
$ rsync -lrtpDvP mouse1 testuser@upload.brainimagelibrary.org:/bil/lz/testuser/abcdef0123456789
sending incremental file list
mouse1/
mouse1/data1.tiff
1356122 100% 126.20MB/s 0:00:00 (xfer#1, to-check=0/2)
sent 1356392 bytes received 35 bytes 2712854.00 bytes/sec
total size is 1356122 speedup is 1.00
Using sftp to upload.brainimagelibrary.org: An example logging into the data transfer node as testuser is shown below. The first cd command is used to move to the BIL landing zone for the submission collection. The mkdir command is used to create a sub-directory (called mouse1 in the landing zone, while the second cd command is used to move to this subdirectory. Finally, the put command is used to upload data - in this case, a single file (data1.tiff):
$ sftp testuser@upload.brainimagelibrary.org
The authenticity of host 'upload.brainimagelibrary.org (128.182.108.164)' can't be established.
ECDSA key fingerprint is 32:cf:46:44:3d:9c:8e:b2:1d:14:03:66:45:0b:11:29.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'upload.brainimagelibrary.org,128.182.108.164' (ECDSA) to the list of known hosts.
testuser@upload.brainimagelibrary.org's password:
Connected to upload.brainimagelibrary.org.
sftp> cd /bil/lz/testuser/abcdef0123456789
sftp> mkdir /bil/lz/testuser/abcdef0123456789/mouse1
sftp> cd /bil/lz/testuser/abcdef0123456789/mouse1
sftp> put data1.tiff
Uploading data1.tiff to /bil/lz/testuser/abcdef0123456789/mouse1/data1.tiff
data1.tiff 0% 0 0.0KB/s --:-- ETA
data1.tiff 100% 1324KB 1.3MB/s 00:00
sftp> exit
Due to their ability to resume interupted transfers, the use of rsync and Globus is recommended over sftp.
Using scp to upload.brainimagelibrary.org: An example uploading data (data1.tiff) to the landing zone as testuser is shown below:
$ scp data1.tiff testuser@upload.brainimagelibrary.org:/bil/lz/abcdef0123456789/mouse1/data1.tiff
testuser@upload.brainimagelibrary.org's password:
data1.tiff 0% 0 0.0KB/s --:-- ETA
data1.tiff 100% 1324KB 1.3MB/s 00:00
Due to their ability to resume interupted transfers, the use of rsync and Globus is recommended over scp.
For uploading data stored on Google Drive, you have the option to use rclone to transfer data directly.
Globus file transfer web service (www.globus.org) can be used to schedule and execute data transfers in a reliable manner. Globus will attempt to resume interrupted file transfers and will send a notification via e-mail upon completion of data transfer tasks. Globus web service is free for non-commercial use.
Data uploads to the BIL landing zone (/bil/lz) via Globus are accomplished via authenticated access to the Globus endpoint “Brain Image Library Upload with CILogon Authentication”.
To use Globus, you can log in with an existing institutional identity, Google, ORCID, or a Globus ID as described at https://docs.globus.org/how-to/get-started/.
For data transfers to or from your local computer, a free Globus Connect Personal client application should be installed, as described at https://www.globus.org/globus-connect-personal.
One-time setup to authenticate via CILogon to BIL for Globus data uploads:
To upload data via Globus to the BIL landing zone (/bil/lz), you must first have a BIL account and then provide the BIL Helpdesk with the CILogon identity information in order to map that CILogon identity to your BIL account for authentication via Globus.
Please provide your CILogon Certificate Subject and ePPN information to the BIL Helpdesk (bil-support@psc.edu) as follows:
Your CILogon Certificate Subject and ePPN will be installed on BIL data transfer servers within one business day, after which you will be able to transfer files via Globus to and from BIL.
Transferring files with Globus
The Globus file transfer service moves data between Globus Connect Servers (formerly GridFTP endpoints). A Globus Connect Server service may be on your own desktop/laptop (Globus Connect Personal, See https://www.globus.org/globus-connect-personal) or installed on a host system connected to data storage at a remote site such as BIL at PSC. File transfers may be scheduled between Globus Connect Servers using the web-based Globus File Transfer service at https://app.globus.org/file-manager, or using a Python-based command-line interface (See https://docs.globus.org/cli/ for installation and examples).
Once all data has been uploaded to the landing zone and the metadata has been submitted, request that the data is validated and made publicly avaliable. This can be done through the submission portal by selecting Submit Publish Request
from the main menu of the submission portal. If an embargo period is being requested, please send email to bil-support@psc.edu along with the submission id and brief note.
If data validation fails, the data submitter will be notified by email. The submitter should address the validation issue(s) and re-submit the validation request. Both data and metadata need to pass the validation checks – datasets that fail the validation process are considered incomplete and will not be made publicly available.
Image File Formats
The ingestion process supports native TIFF and JPEG 2000 image file formats and formats that are convertible to the OME-TIFF format via the bioformats tool. (For more information on the bioformats tool see: https://docs.openmicroscopy.org/bio-formats/5.7.3/formats/index.html.)
Metadata Formats and Schema
Please see the Submission Portal for the most recent metadata specification.
The archive is transitioning to a new, more comprehensive metadata schema developed as a part of the NIH BRAIN Initiative "BRAIN 3D MICROSCOPY STANDARDS PROJECT". For more information about the standards project, please see the Dory website. A draft spreadsheet implementing this standard, which will soon be required by the ingest portal is available here. Collecting metadata on this more extensive form will also enable BIL to issue DOIs for datasets when data is submitted.
Sharing and computing on data prior to submission:
All data submitters are given access to a login node "login.brainimagelibrary.org". To connect to this node, use a terminal program that implements ssh (such as xterm, the native mac terminal program, or PuTTY) and use the same username and password that you would use for the submission portal. The login node is ideal for non-cpu intensive processing, such as editing or deleting a file that is already in the landing zone. The login node is not intended for heavy computation or visualization, but it is connected to the BIL Computational Cluster. The BIL computational cluster provides a suitable resource for both computation and visualization. in addition, if you need to perform more intensive processing on your data, please see this computing and visualization page or contact the BIL Helpdesk for further assistance.
Assistance and Help
If you need assistance with anything, please contact the BIL Helpdesk for assistance. Please note that we are located in the Eastern Time zone.
Please note that BIL also provides networking support. If you are experiencing networking issues related to data transfer (including slow transfer speeds), contact the BIL Helpdesk. If network issues prevent data transfer, we may recommend that you send your datasets to us via an alternate path such on our BrainBall portable device or on LTO tape.
Distribution Licence
Data contributed to the library will be redistributed to others under a Creative Commons Attribution-ShareAlike 4.0 International License. In addition, all data submitters expressly permit the Library data to be transferred to and maintained by another open data repository or by a government agency such as the US National Institutes of Health.Release of entries
Data submitted to the library, by default, will be released as soon as possible once data has completed the submission process. Optionally, data submitters can select a limited embargo period for their submitted data consistant with the data sharing policies of the BRAIN initiative and NIH data sharing policies, or up to one year if the data was not NIH funded.
Assignment of DOI
BIL will soon issue permanent, citable DOI for submitted datasets, when the new metadata schema is implemented. BIL can also issue DOI's for groups of data so that they may be cited in a paper (See here for an example).Changes to entries after submission
Changes may be made to submitted entries prior to release. Minor changes, such as updating metadata or adding citations may be made after dataset release. Major revisions will require the existing entry to be marked as obsolete and replaced by a new entry. The DOI pointing to the metadata for the obsolete entry may have a superseded-by field added to it to point users to the superceded entry. There may be also be circumstances in which an entry in the library may be marked as withdrawn (for example, research misconduct). In those cases, the DOI for the obsolete entry will have a withdrawn field added to it.
Use of Pre-publication Data
The Library expects all users of the submitted data, unless explicitly granted by the data submitter or the data submitter's project, to grant the authors of the data the right to publish the first paper concerning the dataset within three years of deposit. If there is no publication associated with the dataset, please contact the listed authors for permission to publish. There is no restriction on post-publication data.