The following terms in this document have specific context, as defined below:
If you have problems with the steps below, do not hesitate to contact the BIL Helpdesk for assistance.
Follow these instructions to create an XSEDE portal Account.
Send email to email@example.com along with your XSEDE Portal Account. You will receive an email message once access has been enabled. Please allow 24 hours for access to be granted.
Follow these instructions to set your initial password.
Users that do not have PI/Data manager permissions will not have access to
the functions described in this step. If you need access to these functions,
have the PI send email to the BIL helpdesk with your BIL userid and name and request that you be granted this portal permission.
The PI dashboard allows PIs and Data managers to define and manage projects that will be submitting datasets to BIL. This dashboard will allow the PI to add authorized users to the project and see the status of in-progress, unsuccessful, and successful submissions associated with the project, regardless of the users that uploaded and submitted the data.
To enter (or edit) project information, Select the Manage Projects option. From this menu you can create a new project and view both the personnel and submissions associated with each project.
To define a project select the Create a New Project button. Enter the Project Name and the grant number associated with the project in the Funded By: field. If the project is a BICCN project, please be sure to tag the project appropriately and make sure that the Project Name field contains the project name assigned by BCDC.
To add a user to the project (such as a data submitter) select the View Personnel link and add the user.
There MUST be a project defined to create a BIL submission
The next step in the submission process is to create a BIL submission. A submission is an ingestion wrapper that contains one or more related datasets and associated metadata. Submissions will inherit project metadata (such as the NIH project that the data is associated with), thus all datasets within a submission must belong to the same project.
A submission can contain a single or multiple datasets. In general, smaller submissions are recommended because all datasets within a submission must pass the validation process for the datasets within the submission to be published.
When created, each BIL submission will have a a unique 16-digit identifier associated with it (e.g. 6247417d691a4548) and a unique dropbox-like landing zone directory (e.g. /bil/lz/user/6247417d691a4548. This landing zone directory is where the datasets belonging to the submission must be transferred to for validation and ingestion processing.
To create a submission, select the Submission sub-option of the New menu.
Next, enter the required metadata associated with the submission collection (see image below).
When you are done filling out the form, select the Save button.
Metadata for each dataset within a submission collection can be uploaded through the submission portal in spreadsheet format at this time. A template spreadsheet can be found on the submission portal by selecting the New Metadata sub-option of the New menu:
If you are NOT ready to upload metadata at this time, click Cancel
to exit the upload metadata spreadsheet step.
When you are ready to load your metadata,
return to the portal and select the New Metadata sub-option of the New menu.
Note that while preparing the metadata spreadsheet, separate data directories need to be listed for each dataset in the submission. The recommended practices to structure your data directories for submission are described here.
After a metadata spreadsheet is prepared, Select the submission collection it will be associated with. Then select the Upload Metadata button to locate your metadata spreadsheet and upload it.
As mentioned above, the submission portal creates an upload landing zone space for you to transfer your image data to. To find this landing zone, select the Submissions sub-option of the View menu. The field "Data Path" shows the landing zone space for the submission.
The ingestion process currently supports native TIFF and JPEG 2000 image
file formats and will soon support formats that are convertible to the
OME-TIFF format via the
bioformats tool. For more information on the bioformats tool see:
Note that separate data subdirectories are required for each dataset in the submission. For example, if you had an experiment containing 5 mouse datasets that you wanted to to include as a single submission collection, you would create 5 subdirectories in the landing zone for the submission collection, one for each mouse dataset. e.g:/bil/lz/testuser/abcdef0123456789/mouse1
The recommended practices to structure your data directories are further described here.
To tie metadata to an image dataset, each image dataset must be uploaded in a separate subdirectory.
Due to size, image data can not be uploaded through the submission portal (submit.brainimagelibrary.org). It must be uploaded separately through the BIL data transfer nodes, which are available at the virtual host: upload.brainimagelibrary.org. All users authorized to use the data submission portal (submit.brainimagelibrary.org) are also authorized to use the data transfer nodes (upload.brainimagelibrary.org). The username and passwords are the same on both systems.
$ rsync -lrtpDvP mouse1 firstname.lastname@example.org:/bil/lz/testuser/abcdef0123456789 sending incremental file list mouse1/ mouse1/data1.tiff 1356122 100% 126.20MB/s 0:00:00 (xfer#1, to-check=0/2) sent 1356392 bytes received 35 bytes 2712854.00 bytes/sec total size is 1356122 speedup is 1.00
$ sftp email@example.com The authenticity of host 'upload.brainimagelibrary.org (184.108.40.206)' can't be established. ECDSA key fingerprint is 32:cf:46:44:3d:9c:8e:b2:1d:14:03:66:45:0b:11:29. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'upload.brainimagelibrary.org,220.127.116.11' (ECDSA) to the list of known hosts. firstname.lastname@example.org's password: Connected to upload.brainimagelibrary.org. sftp> cd /bil/lz/testuser/abcdef0123456789 sftp> mkdir /bil/lz/testuser/abcdef0123456789/mouse1 sftp> cd /bil/lz/testuser/abcdef0123456789/mouse1 sftp> put data1.tiff Uploading data1.tiff to /bil/lz/testuser/abcdef0123456789/mouse1/data1.tiff data1.tiff 0% 0 0.0KB/s --:-- ETA data1.tiff 100% 1324KB 1.3MB/s 00:00 sftp> exit
Due to their ability to resume interupted transfers, the use of rsync and Globus is recommended over sftp.
$ scp data1.tiff email@example.com:/bil/lz/abcdef0123456789/mouse1/data1.tiff firstname.lastname@example.org's password: data1.tiff 0% 0 0.0KB/s --:-- ETA data1.tiff 100% 1324KB 1.3MB/s 00:00
Due to their ability to resume interupted transfers, the use of rsync and Globus is recommended over scp.
Once all data has been uploaded to the landing zone area and all metadata has been uploaded to through the submission portal, request that the data is validated and made publicly avaliable. This can be done through the submission portal by selecting "Submit Publish request" from the main menu. If an embargo period is being requested, please send email to email@example.com along with the submission id. Please note that the dataset should be embargoed prior to public release, in the email message.
If data fails to validate, the data submitter will be notified by email. The submitter should address the validation issue(s) and re-submit the publication request. Both data and metadata need to pass the validation checks -- datasets that fail the validation process are considered incomplete and will not be made publicly available.
Image File Formats
The ingestion process supports native TIFF and JPEG 2000 image file formats and formats that are convertible to the OME-TIFF format via the bioformats tool. (For more information on the bioformats tool see: https://docs.openmicroscopy.org/bio-formats/5.7.3/formats/index.html.)
Metadata Formats and Schema
Please see the Submission Portal for the most recent metadata specification.
The archive is transitioning to a new, more comprehensive metadata schema developed as a part of the NIH BRAIN Initiative "BRAIN 3D MICROSCOPY STANDARDS PROJECT". For more information about the standards project, please see the Dory website. A draft spreadsheet implementing this standard, which will soon be required by the ingest portal is available here. Collecting metadata on this more extensive form will also enable BIL to issue DOIs for datasets when data is submitted.
Sharing and computing on data prior to submission:
All data submitters are given access to a login node "login.brainimagelibrary.org". To connect to this node, use a terminal program that implements ssh (such as xterm, the native mac terminal program, or PuTTY) and use the same username and password that you would use for the submission portal. The login node is ideal for non-cpu intensive processing, such as editing or deleting a file that is already in the landing zone. The login node is not intended for heavy computation or visualization, but it is connected to the BIL Computational Cluster. The BIL computational cluster provides a suitable resource for both computation and visualization. in addition, if you need to perform more intensive processing on your data, please see this computing and visualization page or contact the BIL Helpdesk for further assistance.
Assistance and Help
If you need assistance with anything, please contact the BIL Helpdesk for assistance. Please note that we are located in the Eastern Time zone.
Please note that BIL also provides networking support. If you are experiencing networking issues related to data transfer (including slow transfer speeds), contact the BIL Helpdesk. If network issues prevent data transfer, we may recommend that you send your datasets to us via an alternate path such on our BrainBall portable device or on LTO tape.
Distribution LicenceData contributed to the library will be redistributed to others under a Creative Commons Attribution-ShareAlike 4.0 International License. In addition, all data submitters expressly permit the Library data to be transferred to and maintained by another open data repository or by a government agency such as the US National Institutes of Health.
Release of entries
Data submitted to the library, by default, will be released as soon as possible once data has completed the submission process. Optionally, data submitters can select a limited embargo period for their submitted data consistant with the data sharing policies of the BRAIN initiative and NIH data sharing policies, or up to one year if the data was not NIH funded.
Assignment of DOIBIL will soon issue permanent, citable DOI for submitted datasets, when the new metadata schema is implemented. BIL can also issue DOI's for groups of data so that they may be cited in a paper (See here for an example).
Changes to entries after submission
Changes may be made to submitted entries prior to release. Minor changes, such as updating metadata or adding citations may be made after dataset release. Major revisions will require the existing entry to be marked as obsolete and replaced by a new entry. The DOI pointing to the metadata for the obsolete entry may have a superseded-by field added to it to point users to the superceded entry. There may be also be circumstances in which an entry in the library may be marked as withdrawn (for example, research misconduct). In those cases, the DOI for the obsolete entry will have a withdrawn field added to it.
Use of Pre-publication Data
The Library expects all users of the submitted data, unless explicitly granted by the data submitter or the data submitter's project, to grant the authors of the data the right to publish the first paper concerning the dataset within three years of deposit. If there is no publication associated with the dataset, please contact the listed authors for permission to publish. There is no restriction on post-publication data.