Details
-
Type:
Task
-
Status: To-Do (View Workflow)
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:
-
Story Points:2
-
Epic Link:
-
Sprint:Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May
Description
Watch this: https://drive.google.com/file/d/1uZHOoVkl-AgQ-jogSGxFH5zulD3wNQeS/view?usp=sharing
The bigwig format is a graph data file format that is supposed to be random access. It contains a "header" at the beginning of the file that maps locations in the file to locations in the genome.
When users open and then click "Load Data" for a small region in the genome, loading the bigwig file's data should feel very fast because IGB only needs to send a byte range request for a small part of the file.
However, this does not seem to be happening.
In fact, the same exact data converted to a different format - tabix-indexed bedgraph - loads much, much faster.
This task is to investigate what is going on. Why is bw file loading so incredibly slow?
Please report what you find as comments in this ticket.
See below comment for example data.
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
| Field | Original Value | New Value |
|---|---|---|
| Epic Link | IGBF-1765 [ 17855 ] |
| Rank | Ranked higher |
| Assignee | Ann Loraine [ aloraine ] |
| Sprint | Summer 2019 Sprint 12 [ 71 ] | Summer 2019 Sprint 12, Fall 2019 Sprint 1 [ 71, 72 ] |
| Rank | Ranked higher |
| Rank | Ranked higher |
| Workflow | Loraine Lab Workflow [ 18662 ] | Fall 2019 Workflow Update [ 19012 ] |
| Sprint | Summer 2019 Sprint 12, Fall 2019 Sprint 1 [ 71, 72 ] | Summer 2019 Sprint 12, Fall 2019 Sprint 2 [ 71, 73 ] |
| Sprint | Summer 2019 Sprint 12, Fall 2019 Sprint 2 [ 71, 73 ] | Summer 2019 Sprint 12, Fall 2019 Sprint 8 [ 71, 78 ] |
| Workflow | Fall 2019 Workflow Update [ 19012 ] | Revised Fall 2019 Workflow Update [ 21123 ] |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Sprint | Summer 2019 Sprint 12, Fall 7 : 11 Nov to 22 Nov [ 71, 78 ] | Summer 2019 Sprint 12 [ 71 ] |
| Rank | Ranked higher |
| Status | In Progress [ 3 ] | To-Do [ 10305 ] |
| Sprint | Summer 2019 Sprint 12 [ 71 ] | Summer 2019 Sprint 12, Spring 2 : 20 Jan to 31 Jan [ 71, 85 ] |
| Rank | Ranked higher |
| Sprint | Summer 2019 Sprint 12, Spring 2 : 20 Jan to 31 Jan [ 71, 85 ] | Summer 2019 Sprint 12, Spring 3 : 3 Feb to 14 Feb [ 71, 86 ] |
| Rank | Ranked lower |
| Rank | Ranked lower |
| Sprint | Summer 2019 Sprint 12, Spring 3 : 3 Feb to 14 Feb [ 71, 86 ] | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 21 Feb 2 [ 71, 87 ] |
| Rank | Ranked higher |
| Sprint | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb [ 71, 87 ] | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar [ 71, 87, 89 ] |
| Rank | Ranked higher |
| Sprint | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar [ 71, 87, 89 ] | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar [ 71, 87, 89, 90 ] |
| Rank | Ranked higher |
| Assignee | Ann Loraine [ aloraine ] |
| Status | To-Do [ 10305 ] | In Progress [ 3 ] |
| Description |
The bigwig format is a graph data file format that is supposed to be random access. It contains a "header" at the beginning of the file that maps locations in the file to locations in the genome.
When users open and then click "Load Data" for a small region in the genome, loading the bigwig file's data should feel very fast because IGB only needs to send a byte range request for a small part of the file. However, this does not seem to be happening. This task is to investigate what is going on. Why is it so slow? Please report what you find as comments in this ticket. Contact Dr. Loraine for some example data. |
The bigwig format is a graph data file format that is supposed to be random access. It contains a "header" at the beginning of the file that maps locations in the file to locations in the genome.
When users open and then click "Load Data" for a small region in the genome, loading the bigwig file's data should feel very fast because IGB only needs to send a byte range request for a small part of the file. However, this does not seem to be happening. In fact, the same exact data converted to a different format - tabix-indexed bedgraph - loads much, much faster. This task is to investigate what is going on. Why is bw file loading so incredibly slow? Please report what you find as comments in this ticket. See below comment for example data. |
| Assignee | Ann Loraine [ aloraine ] |
| Status | In Progress [ 3 ] | To-Do [ 10305 ] |
| Description |
The bigwig format is a graph data file format that is supposed to be random access. It contains a "header" at the beginning of the file that maps locations in the file to locations in the genome.
When users open and then click "Load Data" for a small region in the genome, loading the bigwig file's data should feel very fast because IGB only needs to send a byte range request for a small part of the file. However, this does not seem to be happening. In fact, the same exact data converted to a different format - tabix-indexed bedgraph - loads much, much faster. This task is to investigate what is going on. Why is bw file loading so incredibly slow? Please report what you find as comments in this ticket. See below comment for example data. |
Watch this: https://drive.google.com/file/d/1uZHOoVkl-AgQ-jogSGxFH5zulD3wNQeS/view?usp=sharing
The bigwig format is a graph data file format that is supposed to be random access. It contains a "header" at the beginning of the file that maps locations in the file to locations in the genome. When users open and then click "Load Data" for a small region in the genome, loading the bigwig file's data should feel very fast because IGB only needs to send a byte range request for a small part of the file. However, this does not seem to be happening. In fact, the same exact data converted to a different format - tabix-indexed bedgraph - loads much, much faster. This task is to investigate what is going on. Why is bw file loading so incredibly slow? Please report what you find as comments in this ticket. See below comment for example data. |
| Sprint | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar [ 71, 87, 89, 90 ] | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10 [ 71, 87, 89, 90, 91 ] |
| Rank | Ranked higher |
| Rank | Ranked higher |
| Sprint | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10 [ 71, 87, 89, 90, 91 ] | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 7 : 13 Apr to 24 Apr [ 71, 87, 89, 90, 91, 92 ] |
| Rank | Ranked higher |
| Sprint | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 7 : 13 Apr to 24 Apr [ 71, 87, 89, 90, 91, 92 ] | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May [ 71, 87, 89, 90, 91, 93 ] |
| Rank | Ranked lower |
| Rank | Ranked higher |
| Sprint | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May [ 71, 87, 89, 90, 91, 93 ] | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May, Spring 8 : 11 May to 25 May [ 71, 87, 89, 90, 91, 93, 94 ] |
| Rank | Ranked higher |
| Sprint | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May, Spring 8 : 11 May to 25 May [ 71, 87, 89, 90, 91, 93, 94 ] | Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May [ 71, 87, 89, 90, 91, 93 ] |
| Link | This issue relates to IGBF-2195 [ IGBF-2195 ] |
| Rank | Ranked higher |
| Rank | Ranked higher |
This issue is directly related to IGBF-2954. In that ticket little difference was found in the speed of loading bigWig files. However, with the files mentioned in this ticket and trying to load them with the whole chromosome in view (Chr1:0-30,427,671), I found a significant difference in the speed with which they loaded (the bigWig file loaded significantly slower).
I think it would be a good idea to take another look at the IGB bigWig code to determine where/why the bigWig file is loading slowly.
To test:
Open the A_thaliana_Jun_2009 genome in IGB.
Navigate to Chr1:0-30,427,671
Load the following URL in IGB: https://krizek-lab.s3.amazonaws.com/stage_6and7_chipseq/s6s7-chipseq-ANT_VENUS/s6s7-chip-ANT_VENUS.chipseq.bw
Click Load Data
Note the IGB Log when loading the bigWig file. When I load the bedGraph file there is only a single line in the log.
09:52:59.196 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:03.335 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:07.369 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:11.465 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:15.824 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:20.854 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:25.111 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:29.421 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:33.732 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:38.906 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:43.117 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:47.407 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:51.735 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:53:56.054 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:54:00.505 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:54:05.074 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:54:10.539 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:54:14.906 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:54:19.365 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
09:54:23.448 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
Here is a bigwig file - Arabidopsis thaliana (A_thaliana_Jun_2009) genome:
Here is the tabix-indexed bedgraph version of the same data set:
Note that loading the bigwig file is much slower than the bedgraph file.
(Note: To convert a bigwig file to a bedgraph file, use this Unix utility from Jim Kent: http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/bigWigToBedGraph)