Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-1938

Investigate: Why is bigwig loading so slow?

        Details

        • Type: Task
        • Status: To-Do (View Workflow)
        • Priority: Major
        • Resolution: Unresolved
        • Affects Version/s: None
        • Fix Version/s: None
        • Labels:
        • Story Points:
          2
        • Epic Link:
        • Sprint:
          Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May

          Description

          Watch this: https://drive.google.com/file/d/1uZHOoVkl-AgQ-jogSGxFH5zulD3wNQeS/view?usp=sharing

          The bigwig format is a graph data file format that is supposed to be random access. It contains a "header" at the beginning of the file that maps locations in the file to locations in the genome.

          When users open and then click "Load Data" for a small region in the genome, loading the bigwig file's data should feel very fast because IGB only needs to send a byte range request for a small part of the file.

          However, this does not seem to be happening.

          In fact, the same exact data converted to a different format - tabix-indexed bedgraph - loads much, much faster.

          This task is to investigate what is going on. Why is bw file loading so incredibly slow?

          Please report what you find as comments in this ticket.

          See below comment for example data.

            Attachments

              Issue Links

              relates to

              Task - A task that needs to be done. IGBF-2954 Investigate: why is loading bb (bigbed) and bigwig (bw) files so slow?

              • Major - Major loss of function.
              • Closed

                Activity

                Ascending order - Click to sort in descending order
                ann.loraine Ann Loraine created issue -
                ann.loraine Ann Loraine made changes -
                Field Original Value New Value
                Epic Link IGBF-1765 [ 17855 ]
                ann.loraine Ann Loraine made changes -
                Rank Ranked higher
                ann.loraine Ann Loraine made changes -
                Assignee Ann Loraine [ aloraine ]
                ann.loraine Ann Loraine made changes -
                Sprint Summer 2019 Sprint 12 [ 71 ] Summer 2019 Sprint 12, Fall 2019 Sprint 1 [ 71, 72 ]
                ann.loraine Ann Loraine made changes -
                Rank Ranked higher
                ann.loraine Ann Loraine made changes -
                Rank Ranked higher
                ann.loraine Ann Loraine made changes -
                Workflow Loraine Lab Workflow [ 18662 ] Fall 2019 Workflow Update [ 19012 ]
                ann.loraine Ann Loraine made changes -
                Sprint Summer 2019 Sprint 12, Fall 2019 Sprint 1 [ 71, 72 ] Summer 2019 Sprint 12, Fall 2019 Sprint 2 [ 71, 73 ]
                ann.loraine Ann Loraine made changes -
                Sprint Summer 2019 Sprint 12, Fall 2019 Sprint 2 [ 71, 73 ] Summer 2019 Sprint 12, Fall 2019 Sprint 8 [ 71, 78 ]
                ann.loraine Ann Loraine made changes -
                Workflow Fall 2019 Workflow Update [ 19012 ] Revised Fall 2019 Workflow Update [ 21123 ]
                Status Open [ 1 ] In Progress [ 3 ]
                ann.loraine Ann Loraine made changes -
                Sprint Summer 2019 Sprint 12, Fall 7 : 11 Nov to 22 Nov [ 71, 78 ] Summer 2019 Sprint 12 [ 71 ]
                ann.loraine Ann Loraine made changes -
                Rank Ranked higher
                ann.loraine Ann Loraine made changes -
                Status In Progress [ 3 ] To-Do [ 10305 ]
                ann.loraine Ann Loraine made changes -
                Sprint Summer 2019 Sprint 12 [ 71 ] Summer 2019 Sprint 12, Spring 2 : 20 Jan to 31 Jan [ 71, 85 ]
                ann.loraine Ann Loraine made changes -
                Rank Ranked higher
                ann.loraine Ann Loraine made changes -
                Sprint Summer 2019 Sprint 12, Spring 2 : 20 Jan to 31 Jan [ 71, 85 ] Summer 2019 Sprint 12, Spring 3 : 3 Feb to 14 Feb [ 71, 86 ]
                ann.loraine Ann Loraine made changes -
                Rank Ranked lower
                ann.loraine Ann Loraine made changes -
                Rank Ranked lower
                ann.loraine Ann Loraine made changes -
                Sprint Summer 2019 Sprint 12, Spring 3 : 3 Feb to 14 Feb [ 71, 86 ] Summer 2019 Sprint 12, Spring 3 : 17 Feb to 21 Feb 2 [ 71, 87 ]
                ann.loraine Ann Loraine made changes -
                Rank Ranked higher
                nfreese Nowlan Freese made changes -
                Sprint Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb [ 71, 87 ] Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar [ 71, 87, 89 ]
                nfreese Nowlan Freese made changes -
                Rank Ranked higher
                nfreese Nowlan Freese made changes -
                Sprint Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar [ 71, 87, 89 ] Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar [ 71, 87, 89, 90 ]
                nfreese Nowlan Freese made changes -
                Rank Ranked higher
                ann.loraine Ann Loraine made changes -
                Assignee Ann Loraine [ aloraine ]
                ann.loraine Ann Loraine made changes -
                Status To-Do [ 10305 ] In Progress [ 3 ]
                ann.loraine Ann Loraine made changes -
                Description The bigwig format is a graph data file format that is supposed to be random access. It contains a "header" at the beginning of the file that maps locations in the file to locations in the genome.

                When users open and then click "Load Data" for a small region in the genome, loading the bigwig file's data should feel very fast because IGB only needs to send a byte range request for a small part of the file.

                However, this does not seem to be happening.

                This task is to investigate what is going on. Why is it so slow?

                Please report what you find as comments in this ticket.

                Contact Dr. Loraine for some example data.                 		The bigwig format is a graph data file format that is supposed to be random access. It contains a "header" at the beginning of the file that maps locations in the file to locations in the genome.

                When users open and then click "Load Data" for a small region in the genome, loading the bigwig file's data should feel very fast because IGB only needs to send a byte range request for a small part of the file.

                However, this does not seem to be happening.

                In fact, the same exact data converted to a different format - tabix-indexed bedgraph - loads much, much faster.

                This task is to investigate what is going on. Why is bw file loading so incredibly slow?

                Please report what you find as comments in this ticket.

                See below comment for example data.
                ann.loraine Ann Loraine made changes -
                Assignee Ann Loraine [ aloraine ]
                ann.loraine Ann Loraine made changes -
                Status In Progress [ 3 ] To-Do [ 10305 ]
                ann.loraine Ann Loraine made changes -
                Description The bigwig format is a graph data file format that is supposed to be random access. It contains a "header" at the beginning of the file that maps locations in the file to locations in the genome.

                When users open and then click "Load Data" for a small region in the genome, loading the bigwig file's data should feel very fast because IGB only needs to send a byte range request for a small part of the file.

                However, this does not seem to be happening.

                In fact, the same exact data converted to a different format - tabix-indexed bedgraph - loads much, much faster.

                This task is to investigate what is going on. Why is bw file loading so incredibly slow?

                Please report what you find as comments in this ticket.

                See below comment for example data.                 		Watch this: https://drive.google.com/file/d/1uZHOoVkl-AgQ-jogSGxFH5zulD3wNQeS/view?usp=sharing

                The bigwig format is a graph data file format that is supposed to be random access. It contains a "header" at the beginning of the file that maps locations in the file to locations in the genome.

                When users open and then click "Load Data" for a small region in the genome, loading the bigwig file's data should feel very fast because IGB only needs to send a byte range request for a small part of the file.

                However, this does not seem to be happening.

                In fact, the same exact data converted to a different format - tabix-indexed bedgraph - loads much, much faster.

                This task is to investigate what is going on. Why is bw file loading so incredibly slow?

                Please report what you find as comments in this ticket.

                See below comment for example data.
                nfreese Nowlan Freese made changes -
                Sprint Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar [ 71, 87, 89, 90 ] Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10 [ 71, 87, 89, 90, 91 ]
                nfreese Nowlan Freese made changes -
                Rank Ranked higher
                ann.loraine Ann Loraine made changes -
                Rank Ranked higher
                ann.loraine Ann Loraine made changes -
                Sprint Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10 [ 71, 87, 89, 90, 91 ] Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 7 : 13 Apr to 24 Apr [ 71, 87, 89, 90, 91, 92 ]
                ann.loraine Ann Loraine made changes -
                Rank Ranked higher
                ann.loraine Ann Loraine made changes -
                Sprint Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 7 : 13 Apr to 24 Apr [ 71, 87, 89, 90, 91, 92 ] Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May [ 71, 87, 89, 90, 91, 93 ]
                ann.loraine Ann Loraine made changes -
                Rank Ranked lower
                ann.loraine Ann Loraine made changes -
                Rank Ranked higher
                ann.loraine Ann Loraine made changes -
                Sprint Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May [ 71, 87, 89, 90, 91, 93 ] Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May, Spring 8 : 11 May to 25 May [ 71, 87, 89, 90, 91, 93, 94 ]
                ann.loraine Ann Loraine made changes -
                Rank Ranked higher
                ann.loraine Ann Loraine made changes -
                Sprint Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May, Spring 8 : 11 May to 25 May [ 71, 87, 89, 90, 91, 93, 94 ] Summer 2019 Sprint 12, Spring 3 : 17 Feb to 28 Feb, Spring 4 : 2 Mar to 13 Mar, Spring 5 : 16 Mar to 27 Mar, Spring 6 : 30 Mar to Apr 10, Spring 8 : 24 Apr to 8 May [ 71, 87, 89, 90, 91, 93 ]
                ann.loraine Ann Loraine made changes -
                Link This issue relates to IGBF-2195 [ IGBF-2195 ]
                ann.loraine Ann Loraine made changes -
                Rank Ranked higher
                nfreese Nowlan Freese made changes -
                Link This issue relates to IGBF-2954 [ IGBF-2954 ]
                nfreese Nowlan Freese made changes -
                Rank Ranked higher

                  People

                  • Assignee:
                    Unassigned
                    Reporter:
                    ann.loraine Ann Loraine
                  • Votes:
                    0 Vote for this issue
                    Watchers:
                    Start watching this issue

                    Dates

                    • Created:
                      Updated: