Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-1938

Investigate: Why is bigwig loading so slow?

        Details

        • Type: Task
        • Status: To-Do (View Workflow)
        • Priority: Major
        • Resolution: Unresolved
        • Affects Version/s: None
        • Fix Version/s: None
        • Labels:
        • Story Points:
          2
        • Epic Link:
        • Sprint:
          Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May

          Description

          Watch this: https://drive.google.com/file/d/1uZHOoVkl-AgQ-jogSGxFH5zulD3wNQeS/view?usp=sharing

          The bigwig format is a graph data file format that is supposed to be random access. It contains a "header" at the beginning of the file that maps locations in the file to locations in the genome.

          When users open and then click "Load Data" for a small region in the genome, loading the bigwig file's data should feel very fast because IGB only needs to send a byte range request for a small part of the file.

          However, this does not seem to be happening.

          In fact, the same exact data converted to a different format - tabix-indexed bedgraph - loads much, much faster.

          This task is to investigate what is going on. Why is bw file loading so incredibly slow?

          Please report what you find as comments in this ticket.

          See below comment for example data.

            Attachments

              Issue Links

              relates to

              Task - A task that needs to be done. IGBF-2954 Investigate: why is loading bb (bigbed) and bigwig (bw) files so slow?

              • Major - Major loss of function.
              • Closed

                Activity

                Ascending order - Click to sort in descending order
                Hide
                Permalink
                ann.loraine Ann Loraine added a comment - - edited

                Here is a bigwig file - Arabidopsis thaliana (A_thaliana_Jun_2009) genome:

                Here is the tabix-indexed bedgraph version of the same data set:

                Note that loading the bigwig file is much slower than the bedgraph file.

                (Note: To convert a bigwig file to a bedgraph file, use this Unix utility from Jim Kent: http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/bigWigToBedGraph)

                Show
                ann.loraine Ann Loraine added a comment - - edited Here is a bigwig file - Arabidopsis thaliana (A_thaliana_Jun_2009) genome: https://krizek-lab.s3.amazonaws.com/stage_6and7_chipseq/s6s7-chipseq-ANT_VENUS/s6s7-chip-ANT_VENUS.chipseq.bw Here is the tabix-indexed bedgraph version of the same data set: https://krizek-lab.s3.amazonaws.com/stage_6and7_chipseq/s6s7-chipseq-ANT_VENUS/s6s7-chip-ANT_VENUS.chipseq.bedgraph.gz Note that loading the bigwig file is much slower than the bedgraph file. (Note: To convert a bigwig file to a bedgraph file, use this Unix utility from Jim Kent: http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/bigWigToBedGraph )
                Hide
                Permalink
                nfreese Nowlan Freese added a comment - - edited

                This issue is directly related to IGBF-2954. In that ticket little difference was found in the speed of loading bigWig files. However, with the files mentioned in this ticket and trying to load them with the whole chromosome in view (Chr1:0-30,427,671), I found a significant difference in the speed with which they loaded (the bigWig file loaded significantly slower).

                I think it would be a good idea to take another look at the IGB bigWig code to determine where/why the bigWig file is loading slowly.

                To test:
                Open the A_thaliana_Jun_2009 genome in IGB.
                Navigate to Chr1:0-30,427,671
                Load the following URL in IGB: https://krizek-lab.s3.amazonaws.com/stage_6and7_chipseq/s6s7-chipseq-ANT_VENUS/s6s7-chip-ANT_VENUS.chipseq.bw
                Click Load Data

                Note the IGB Log when loading the bigWig file. When I load the bedGraph file there is only a single line in the log.

                09:52:59.196 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:03.335 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:07.369 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:11.465 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:15.824 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:20.854 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:25.111 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:29.421 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:33.732 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:38.906 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:43.117 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:47.407 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:51.735 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:53:56.054 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:54:00.505 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:54:05.074 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:54:10.539 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:54:14.906 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:54:19.365 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store
                09:54:23.448 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store

                Show
                nfreese Nowlan Freese added a comment - - edited This issue is directly related to IGBF-2954 . In that ticket little difference was found in the speed of loading bigWig files. However, with the files mentioned in this ticket and trying to load them with the whole chromosome in view (Chr1:0-30,427,671), I found a significant difference in the speed with which they loaded (the bigWig file loaded significantly slower). I think it would be a good idea to take another look at the IGB bigWig code to determine where/why the bigWig file is loading slowly. To test: Open the A_thaliana_Jun_2009 genome in IGB. Navigate to Chr1:0-30,427,671 Load the following URL in IGB: https://krizek-lab.s3.amazonaws.com/stage_6and7_chipseq/s6s7-chipseq-ANT_VENUS/s6s7-chip-ANT_VENUS.chipseq.bw Click Load Data Note the IGB Log when loading the bigWig file. When I load the bedGraph file there is only a single line in the log. 09:52:59.196 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:03.335 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:07.369 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:11.465 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:15.824 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:20.854 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:25.111 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:29.421 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:33.732 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:38.906 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:43.117 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:47.407 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:51.735 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:53:56.054 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:54:00.505 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:54:05.074 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:54:10.539 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:54:14.906 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:54:19.365 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store 09:54:23.448 INFO c.a.igb.util.IGBTrustManager - Authenticated CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US,CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE, certificates using default trust store

                  People

                  • Assignee:
                    Unassigned
                    Reporter:
                    ann.loraine Ann Loraine
                  • Votes:
                    0 Vote for this issue
                    Watchers:
                    2 Start watching this issue

                    Dates

                    • Created:
                      Updated: