Implement CyVerse metadata bridge with IGB bookmarks

The CyVerse metadata bridge has been implemented with shortcomings. At the IGB loaded initially, the metadata is sent to IGB when the View In IGB button is clicked,however the foreground and background colors are not rendered then. On clicking it a second time, the colors are rendered when chromosomes are changed. The same happens when User preferences are rest. This happens because the lamda function at line 284 of BookmarkUnibrowControlServlet is skipped. The colors are not rendered when the function, BookmarkController.applyProperties(igbService, seq, parameters, feature, combos), is not rendered, which happens when IGB is initially loaded and when preferences are reset. Further investigation required in this regard.

I have been investigating the issue related to the foreground and background colors not appearing correctly in IGB when sent via CyVerse metadata bridge.

The cause of the issue is that IGB Bookmarks is not capable of loading sym foreground/background colors without first loading the data. For example, the following URL includes the sym color information, but the color will never be shown as no region of the genome is specified to load (so the track is added to IGB, but no data is ever loaded).

//This URL will not show the specified sym color in IGB
http://127.0.0.1:7085/IGBControl?
version=H_sapiens_Dec_2013&
loadresidues=false&
feature_url_0=https://data.cyverse.org/dav-anon/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bed_HomoSapiensChr1.bed&
sym_method_0=https://data.cyverse.org/dav-anon/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bed_HomoSapiensChr1.bed&
sym_col_0=0x00FF00&
sym_bg_0=0xFF0000&
sym_name_0=RefSeq+Curated&
query_url=https://data.cyverse.org/dav-anon/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bed_HomoSapiensChr1.bed&
server_url=

If a chromosome and start/end are added to the above URL, then the specified sym colors will show correctly in IGB as the data will be loaded for that chromosome and region:
seqid=chr1&
start=0&
end=248956422&

There is currently no way to save a bookmark and include foreground/background colors without first loading some amount of data.
For example, in IGB:
Add a file to IGB via Open File... or Open URL... (can use https://data.cyverse.org/dav-anon/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bed_HomoSapiensChr1.bed as an example)
DO NOT click Load Data
In the Data Management Table, change the foreground/background colors
In the Bookmarks tab, create a bookmark, then export the bookmark
Reset Preferences to Default in IGB
Delete the .igb folder
Start IGB
Import bookmark
Load bookmark

The bookmark will not have the correct foreground/background color (default is blue foreground and white background). This can also be seen by Copying the bookmark as a URL and examining the URL. No sym_col or sym_bg will be included in the URL so the default colors will be selected.

Note: Neither Galaxy or BAR specify foreground/background color within the URL.

I'm guessing there is a reason for why data has to be loaded before foreground/background color properties can be applied. However, as we cannot make assumptions about the data from CyVerse, it is not possible to load data coming from CyVerse. We will need to find a workaround for specifying foreground/background color for CyVerse without having to load the data.

Quick comment about BAR:

• BAR site links contain the name of a gene and point to bar.html on the BioViz site.
• Javascript running in bar.html uses a REST service to look up the gene location.
• The javascript code then hits an IGB localhost endpoint that consumes the gene location and also the URL of the Quickload site which hosts the BAR RNA-Seq data. Thanks to this, IGB is able to load the data and use the same names, colors, etc. as are configured for the Quickload site.

Update:
Testing on a bed file I figured out part of the problem. In BookmarkUnibrowControlServlet.java, if the bookmark URL does not specify a chromosome and region, then the first chromosome is selected and some arbitrary logic is applied to pick a starting position to view within that chromosome. However, because the code to pick a starting position is arbitrary, the region in view can be quite small. After digging through the code for parsing the bed this small view seems to be having an unintended effect. In QuickLoadSymLoader.java within the loadAndAddSymmetries method, there is the following line:

List<? extends SeqSymmetry> results = getRegion(span);

The method getRegion(span) eventually calls the BedSymloader.java parseLine method. Though the bed file is not loaded to view in IGB, the contents of at least the specified chromosome are parsed in their entirety. However, only the bed annotations that are within the specified view position are added to a list of syms. If the view is too narrow/small, then nothing is added to the list of syms. This causes the results variable in QuickLoadSymLoader.java loadAndAddSymmetries method to be empty. This eventually leads to the BioSeq.java addAnnotation method assigning type_id2sym to also be empty. BookmarkUnibrowControlServelet.java calls the applyProperties method for BookmarkController.java. BookmarkController applyProperties method tries to parse the annotation type (GraphSym or TypeContainerAnnot), however because no syms were added in the parseLine method of BedSymloader.java, the annotation type held within type_id2sym is empty, and so the bookmark properties such as color are not applied.

Note that this may only affect bed files, additional testing is required on graph and bam files. It also appears that by forcing the view to be the whole chromosome, this also loads all of the data within that region, which is not what we want.

Test URL:
http://127.0.0.1:7085/IGBControl?
version=H_sapiens_Dec_2013&
loadresidues=false&
feature_url_0=https://data.cyverse.org/dav-anon/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bed_HomoSapiensChr1.bed&
sym_method_0=https://data.cyverse.org/dav-anon/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bed_HomoSapiensChr1.bed&
sym_col_0=0x00FF00&
sym_bg_0=0xFF0000&
sym_name_0=RefSeq+Curated&
query_url=https://data.cyverse.org/dav-anon/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bed_HomoSapiensChr1.bed&
server_url=

A side comment which might be relevant: It seems a bit weird that a method from QuickLoadSymLoader.java is being used because at least in this case, there is no Quickload site involved. There is an empty "server_url" parameter in the link, which means that a Quickload server could be involved, however. So maybe it is not so weird, after all?

Karthik Raveendran

When no genome and version are selected for a file in BioViz Connect, the URL that is sent to IGB needs to contain:

version=Custom+Genome+1

This allows the current bookmark logic for custom genomes to be used. Currently if no genome and version are selected in BioViz Connect then the version parameter is empty.

Karthik Raveendran

I have pushed the changes to my branch (https://bitbucket.org/nfreese/nowlanfork-igb/branch/IGBF-2133-CyVerseMetadataBridge).

Fixes: Colors for foreground/background should now render correctly from BioViz Connect. Genome versions or custom genomes should now load correctly.

To test color:
Change foreground/background color and set species/version in BioViz Connect (Bed_HomoSapiensChr1.bed in SmokeTesting).
Click View in IGB and look for the correct genome to load and the foreground/background colors to appear.

To test custom genomes:
Use a file not associated with a genome in BioViz Connect (Bed_HomoSapiensChr1_TEST.bed in SmokeTesting).
Have IGB in the home screen.
Click View in IGB in BioViz Connect.
IGB should open a custom genome.

Use a file not associated with a genome in BioViz Connect (Bed_HomoSapiensChr1_TEST.bed in SmokeTesting).
Click on the human genome in IGB.
Click View in IGB in BioViz Connect.
IGB should load the data in the human genome.

Note: We need to make two changes to BioViz Connect code for the above IGB code to work correctly - this now has its own issue IGBF-2272.
1: If a file is not associated with a genome, the version should be set as "?". It doesn't matter what this is set to, but this is the convention that Galaxy uses.
2: The server_url needs to be set to "cyverse". It doesn't matter what this is set to, but this is the convention that Galaxy uses.

These urls can also be used to test:

http://127.0.0.1:7085/IGBControl?
version=?&
loadresidues=false&
cyverse_data=true&
feature_url_0=http://igbquickload.org/smokeTestingQuickload/H_sapiens_Dec_2013/Bed/Bed_HomoSapien.bed&
sym_method_0=http://igbquickload.org/smokeTestingQuickload/H_sapiens_Dec_2013/Bed/Bed_HomoSapien.bed&
sym_col_0=0x0F0F0F&
sym_bg_0=0x0FFFF0&
sym_name_0=RefSeq+Curated&
query_url=http://igbquickload.org/smokeTestingQuickload/H_sapiens_Dec_2013/Bed/Bed_HomoSapien.bed&
server_url=cyverse

http://127.0.0.1:7085/IGBControl?
version=H_sapiens_Dec_2013&
loadresidues=false&
cyverse_data=true&
feature_url_0=http://igbquickload.org/smokeTestingQuickload/H_sapiens_Dec_2013/Bed/Bed_HomoSapien.bed&
sym_method_0=http://igbquickload.org/smokeTestingQuickload/H_sapiens_Dec_2013/Bed/Bed_HomoSapien.bed&
sym_col_0=0x00FF00&
sym_bg_0=0xFF0000&
sym_name_0=RefSeq+Curated&
query_url=http://igbquickload.org/smokeTestingQuickload/H_sapiens_Dec_2013/Bed/Bed_HomoSapien.bed&
server_url=cyverse

Updated branch with changes to address server_url issue when a genome is specified: https://bitbucket.org/nfreese/nowlanfork-igb/branch/IGBF-2133-CyVerseMetadataBridge

Ready for testing.

Link to pull request: https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/773/igbf-2133-render-color-for-bioviz-connect/diff

Tested the ticket:
Steps involved in Testing:
1) Open IGB and do not load any data. Change the foreground, background, Genome version at Biovizconnect and click on "view in igb" and IGB should load the data with foreground, background, and genome version loaded correctly in the data tracks and bookmarks.
2) Open IGB and load human species data. Change the foreground, background, Genome version at Biovizconnect and click on "view in igb" and IGB should load the data with foreground, background and genome version loaded correctly in the data tracks and bookmarks.
3) Open IGB and load human species data. Change the foreground, background and put Genome version blank or "version" and click on "view in igb" and IGB should load the data with foreground, background with the new values and genome version with the previously loaded values.
4) Also tested with the url's mentioned above in the comments.

Merged and ready for testing.

I have tested the ticket with all the file types in SmokeTesting Folder such as ( bedgraph, bai, bam, tz, gz, bigWig, broadPeak, narrowPeak, wig) and other file types as well. IGB is loading the updated foreground, background and genome versions correctly.
But, If the file name has a space for example (BigBed_Homo Sapien.bigbed) IGB fails to load the file, it shows the following error message.

WARNING: Error reading file header for https://data.cyverse.org/dav-anon/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/BigBed_Homo Sapien.bigbed
java.lang.RuntimeException: Error reading file header for https://data.cyverse.org/dav-anon/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/BigBed_Homo Sapien.bigbed

However, the above version is a Legacy issue which is also present in the previous versions.