Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-2761

Update GIAB annots.xml to point to S3

    Details

    • Type: Improvement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      Situation: The GIAB quickload currently points towards data stored in the GIAB ftp. For some reason it is no longer allowing ftp connections.

      Task: Update the GIAB quickload to point to data stored in the GIAB S3 buckets.

        Attachments

        1. giab_s3_urls.zip
          2.68 MB
        2. Image1.png
          Image1.png
          394 kB
        3. Image2.png
          Image2.png
          737 kB
        4. Image3.png
          Image3.png
          283 kB
        5. Image4.png
          Image4.png
          277 kB
        6. Image5.png
          Image5.png
          253 kB
        7. Image6.png
          Image6.png
          86 kB

          Issue Links

            Activity

            Hide
            nfreese Nowlan Freese added a comment -

            To test in IGB:

            Add this link as a new Quickload under the Data Access tab in Available Data - Configure:
            https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.1/quickload/

            Open the H_sapiens_Dec_2013 human genome.
            Navigate to chr1:80,644,218-80,644,359
            Select one of the Quickload files and click Load Data.
            Look to see that the data load and no errors are thrown.
            Repeat for every Quickload file.

            Open the H_sapiens_Feb_2009 human genome.
            Navigate to chr1:77,876,660-77,876,761
            Select one of the Quickload files and click Load Data.
            Look to see that the data load and no errors are thrown.
            Repeat for every Quickload file.

            Show
            nfreese Nowlan Freese added a comment - To test in IGB: Add this link as a new Quickload under the Data Access tab in Available Data - Configure: https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.1/quickload/ Open the H_sapiens_Dec_2013 human genome. Navigate to chr1:80,644,218-80,644,359 Select one of the Quickload files and click Load Data. Look to see that the data load and no errors are thrown. Repeat for every Quickload file. Open the H_sapiens_Feb_2009 human genome. Navigate to chr1:77,876,660-77,876,761 Select one of the Quickload files and click Load Data. Look to see that the data load and no errors are thrown. Repeat for every Quickload file.
            Hide
            ssegu Sai Supreeth Segu (Inactive) added a comment - - edited

            I have tested each quickload file in the 2 scenarios.

            When I tested in the first scenario using H_sapiens_Dec_2013 human genome, part of the data was not loaded and mostly they are Soft-clips.
            I have added 5 images for that scenario and log for the error is:

            15:31:13.752 WARN o.l.i.quickload.util.QuickloadUtils - Optional species.txt could not be loaded from: https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.1/quickload/species.txt
            15:31:13.826 INFO o.l.i.appstore.IgbAppServerLauncher - Started REST endpoint.

            When I tested in the first scenario using H_sapiens_Dec_2009 human genome, this issue is not there but in AshkenazimTrio --> Son --> Illumina_6kb_matepair was not loaded and log for the error is:

            17:31:38.694 INFO c.a.igb.view.load.GeneralLoadUtils - Loaded RefGene in 2.188 s
            org.lorainelab.igb.bam.BAM$BamIndexNotFoundException: Could not find Bam Index File.
            at org.lorainelab.igb.bam.BAM.findIndexFile(BAM.java:433)
            at org.lorainelab.igb.bam.BAM.getBamIndexUriStr(BAM.java:139)
            at org.lorainelab.igb.bam.BAM.getSAMFileReader(BAM.java:98)
            at org.lorainelab.igb.bam.BAM.init(BAM.java:157)
            at org.lorainelab.igb.bam.XAM.getChromosomeList(XAM.java:100)
            at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadAndAddSymmetries(QuickLoadSymLoader.java:153)
            at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadSymmetriesThread(QuickLoadSymLoader.java:139)
            at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadFeatures(QuickLoadSymLoader.java:119)
            at com.affymetrix.igb.view.load.GeneralLoadUtils.loadFeaturesForSym(GeneralLoadUtils.java:751)
            at com.affymetrix.igb.view.load.GeneralLoadUtils$2.runInBackground(GeneralLoadUtils.java:693)
            at com.affymetrix.igb.view.load.GeneralLoadUtils$2.runInBackground(GeneralLoadUtils.java:688)
            at com.affymetrix.genometry.thread.CThreadWorker.doInBackground(CThreadWorker.java:73)
            at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at javax.swing.SwingWorker.run(SwingWorker.java:334)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)

            cc: Dr.Nowlan Freese

            Show
            ssegu Sai Supreeth Segu (Inactive) added a comment - - edited I have tested each quickload file in the 2 scenarios. When I tested in the first scenario using H_sapiens_Dec_2013 human genome, part of the data was not loaded and mostly they are Soft-clips. I have added 5 images for that scenario and log for the error is: 15:31:13.752 WARN o.l.i.quickload.util.QuickloadUtils - Optional species.txt could not be loaded from: https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.1/quickload/species.txt 15:31:13.826 INFO o.l.i.appstore.IgbAppServerLauncher - Started REST endpoint. When I tested in the first scenario using H_sapiens_Dec_2009 human genome, this issue is not there but in AshkenazimTrio --> Son --> Illumina_6kb_matepair was not loaded and log for the error is: 17:31:38.694 INFO c.a.igb.view.load.GeneralLoadUtils - Loaded RefGene in 2.188 s org.lorainelab.igb.bam.BAM$BamIndexNotFoundException: Could not find Bam Index File. at org.lorainelab.igb.bam.BAM.findIndexFile(BAM.java:433) at org.lorainelab.igb.bam.BAM.getBamIndexUriStr(BAM.java:139) at org.lorainelab.igb.bam.BAM.getSAMFileReader(BAM.java:98) at org.lorainelab.igb.bam.BAM.init(BAM.java:157) at org.lorainelab.igb.bam.XAM.getChromosomeList(XAM.java:100) at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadAndAddSymmetries(QuickLoadSymLoader.java:153) at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadSymmetriesThread(QuickLoadSymLoader.java:139) at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadFeatures(QuickLoadSymLoader.java:119) at com.affymetrix.igb.view.load.GeneralLoadUtils.loadFeaturesForSym(GeneralLoadUtils.java:751) at com.affymetrix.igb.view.load.GeneralLoadUtils$2.runInBackground(GeneralLoadUtils.java:693) at com.affymetrix.igb.view.load.GeneralLoadUtils$2.runInBackground(GeneralLoadUtils.java:688) at com.affymetrix.genometry.thread.CThreadWorker.doInBackground(CThreadWorker.java:73) at javax.swing.SwingWorker$1.call(SwingWorker.java:295) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at javax.swing.SwingWorker.run(SwingWorker.java:334) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) cc: Dr. Nowlan Freese
            Hide
            nfreese Nowlan Freese added a comment -

            I'm not seeing any issues with the 2013 Human genome from the log or images 1-5.

            The AshkenazimTrio --> Son --> Illumina_6kb_matepair was pointing at the wrong bam file. I have updated the annots.xml and it is now working correctly.

            Next step: I need to update igbDefaultPrefs.json

            Show
            nfreese Nowlan Freese added a comment - I'm not seeing any issues with the 2013 Human genome from the log or images 1-5. The AshkenazimTrio --> Son --> Illumina_6kb_matepair was pointing at the wrong bam file. I have updated the annots.xml and it is now working correctly. Next step: I need to update igbDefaultPrefs.json
            Hide
            nfreese Nowlan Freese added a comment -

            A number of comments are included in this declined pull request.

            I think the least amount of work for us would be to leave igbDefaultPrefs.json pointed at the 1.0.0 branch. If we need to update a link in the annots.xml, the change will be made to the 1.0.0 branch, but no new branch will be created. This way all versions of IGB will have the most up to date Quickload info and there will still be a record of the changes (commit history for that branch). If at some point in the future we create a new Quickload format then we could create a new branch and subsequent versions of IGB would point at the new branch.

            Show
            nfreese Nowlan Freese added a comment - A number of comments are included in this declined pull request . I think the least amount of work for us would be to leave igbDefaultPrefs.json pointed at the 1.0.0 branch. If we need to update a link in the annots.xml, the change will be made to the 1.0.0 branch, but no new branch will be created. This way all versions of IGB will have the most up to date Quickload info and there will still be a record of the changes (commit history for that branch). If at some point in the future we create a new Quickload format then we could create a new branch and subsequent versions of IGB would point at the new branch.
            Hide
            nfreese Nowlan Freese added a comment -

            All of the changes have been merged into the release-v1.0.0 branch. No changes need to be made to IGB. I have tested that AshkenazimTrio --> Son --> Illumina_6kb_matepair works in IGB 9.1.8.

            Closing issue.

            Show
            nfreese Nowlan Freese added a comment - All of the changes have been merged into the release-v1.0.0 branch. No changes need to be made to IGB. I have tested that AshkenazimTrio --> Son --> Illumina_6kb_matepair works in IGB 9.1.8. Closing issue.

              People

              • Assignee:
                nfreese Nowlan Freese
                Reporter:
                nfreese Nowlan Freese
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: