I've confirmed that I can visualize this CRAM data in IGB 10.1.0 (although the VCF file doesn't seem to be loading). Although it's unclear if Nebula Genomics was the genome re-sequencing company that produced this data, I think we should move forward with deploying it and making it available as an IGB Quickload ahead of the ASHG conference this Fall.

Passing this ticket to Nowlan Freese for next steps.

Question for Paige Kulzer:

• Did you have to download the entire CRAM file to view it in IGB?

Question for Nowlan Freese:

• Can we make an annots.xml file that points to the remote CRAM file using the third-party provided link and point to the .crai file hosted at another location? I think this might be possible to do with .bai files. I do not know if I am remembering correctly, however. If we can do it with .bai files, it might also be possible with .crai files.

I tried using the link above to open the .cram file in IGB, but that only loaded an empty track. So, yes, the .cram file had to be downloaded so that the .crai file could be created locally before viewing the data in IGB.

Ann Loraine - I'm pretty sure we can specify a separate location for the index file in the annots.xml, but I will double-check and update here. For the third-party provided link, are you referring to the link provided by Nebula genomics? That link is only "live" for a couple of days/weeks before the data are archived and no longer available. Once the data are archived the user has to request the data through their Nebula account. It takes a couple of days before the data are made available again.

reply: No, I am not referring to the Nebula Genomics download link. I meant the link in this ticket to the cram file. If we can refer to it in an annots.xml then we will not have to host it ourselves.

Nowlan - I have the index, I can put it in BioViz Connect and then create an annots.xml and see if I can point it at this cram and the index in BioViz Connect.

Ann Loraine - I was able to create a quickload using the link to the original CRAM file and a link to the CRAM index that I placed in CyVerse (see quickload.zip). Data loaded quickly and I did not see any errors in the log.

From Paige:

I believe you can use samtools to do this! I think this might work: samtools view -Co out.chr1.cram in.cram chr1
To install samtools on a mac, I used Homebrew.
To install Homebrew: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
To install samtools: brew install samtools

From Nowlan:

I think the samtools command would be something like:

samtools view -h -C -T GENOME.fa -o NAME_OF_OUTPUT_FILE.cram NAME_OF_CRAM_INPUT_FILE.cram NAME_OF_CHROMOSOME_TO_EXTRACT

samtools view -h -C -T hg38.fa -o nowlan_nebula_chr1.cram nowlan_nebula_full.cram chr1

I'm pretty sure you need the "-T" and the fasta file for the genome associated with the cram file. See the samtools documentation here: https://www.htslib.org/doc/samtools-view.html

Suggestions:

• Use Quickload Builder App to create an Individual Genome Sequencing repository
• It could have two datasets: chromosome 1 from https://e4d05c260dbc25ea5821e03eb009a2b7-26238.collections.ac2it.arvadosapi.com/_/NB72462M.cram
• Chromosome 1 from a more recent Nebula data set
• Document how you did it in a Jira ticket with the intention to transform the ticket comments into one or more publications, e.g., a Bioinformatics Application Note or slidedeck for a talk
• Make one or two postcards with visually appealing images showing off what a personal genome sequence can look like in IGB; hand them out to audience members or others who would be interested in learning about individual genome sequencing

Add the link to the nebula genomics CRAM file to IGB Quickload server (via subversion) in a new folder "Individual genome sequencing".

Can add cram index to repo, add some kind of documentation explaining where the CRAM file came from and additional details. Maybe update readme or add a txt?

Ann Loraine:

• I installed SVN on my new machine, but the subversion repository appears to be down again and is responding with svn: E200029: Couldn't perform atomic initialization
• We had previously discussed some changes to the annots.xml, but I don't think the changes were saved/updated in the ticket (I'm pretty sure the attached zip is the old/original version). I have added an updated version with what I can remember we discussed for review.
  • Note that the url listed in the annots.xml will be updated to point at whatever documentation we create about the source of the CRAM file. We will also need to update the location of the index.

I think we should bump this ticket down a couple of sprints. This file will not load in IGB 10.0.1 so we will need to do the IGB 10.1.0 release first. I think it would be confusing for users if a new dataset appeared in the default data for humans but wouldn't load.

Discussed with Dr. Loraine, she suggested adding the changes to annots.xml but having it commented out.

Very sorry for blocking your work. I restarted the svn site and reattached the virtual hard drive storing the data.
Logged in using ec2-user@svn.bioviz.org, then did:

[elastic svn ~]$ sudo su
[root@ip-172-31-63-94 ec2-user]# ls /svn
[root@ip-172-31-63-94 ec2-user]# mount -t ext4 /dev/xvdf /svn
[root@ip-172-31-63-94 ec2-user]# ls /svn
genomes  lost+found  svn-auth
[root@ip-172-31-63-94 ec2-user]# 

Follow-up regarding suggestion of commentint-out new data sets in Quickload main:

• I am not actually sure IGB can respect comments in an annots.xml file.

Some additional notes:

• Should be able to comment out the new file in the annots.xml (see stackoverflow).
• It looks like we will need to store the index somewhere online that we can provide a full URL to. I have tried multiple versions of the quickload using bam/cram files. The only time the annots.xml index attribute appears to work is if a full URL path is provided for both the name and index attributes. This doesn't line up with our current annots.xml documentation, but does seem to line up with the requirements for the IGBF-445 ticket where the work was done (links to files and their indexes stored online in different locations). Note that the commit for this work was quite extensive, so it is unclear how easy it would be to improve the annots.xml index attribute logic. I also could not find any examples in our current quickloads where we use the index attribute.
• See my attached annots.xml for examples of what I tested, the only files that did not throw an error regarding the index attribute were "NB72462M - Personal Genome Project" and "thing4".

Error:

java.lang.IllegalArgumentException: URI is not absolute
	at java.base/java.net.URL.of(URL.java:862)
	at java.base/java.net.URI.toURL(URI.java:1172)
	at org.lorainelab.igb.bam.BAM.getSAMFileReader(BAM.java:106)
	at org.lorainelab.igb.bam.BAM.init(BAM.java:160)
	at org.lorainelab.igb.bam.XAM.getChromosomeList(XAM.java:100)
	at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadAndAddSymmetries(QuickLoadSymLoader.java:153)
	at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadSymmetriesThread(QuickLoadSymLoader.java:139)
	at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadFeatures(QuickLoadSymLoader.java:119)
	at com.affymetrix.igb.view.load.GeneralLoadUtils.loadFeaturesForSym(GeneralLoadUtils.java:768)
	at com.affymetrix.igb.view.load.GeneralLoadUtils$2.runInBackground(GeneralLoadUtils.java:710)
	at com.affymetrix.igb.view.load.GeneralLoadUtils$2.runInBackground(GeneralLoadUtils.java:705)
	at com.affymetrix.genometry.thread.CThreadWorker.doInBackground(CThreadWorker.java:73)
	at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:305)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:342)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Added quickload_v3.zip which includes the IGBF-3841 data.