Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-2761

Update GIAB annots.xml to point to S3

    Details

    • Type: Improvement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      Situation: The GIAB quickload currently points towards data stored in the GIAB ftp. For some reason it is no longer allowing ftp connections.

      Task: Update the GIAB quickload to point to data stored in the GIAB S3 buckets.

        Attachments

        1. giab_s3_urls.zip
          2.68 MB
        2. Image1.png
          Image1.png
          394 kB
        3. Image2.png
          Image2.png
          737 kB
        4. Image3.png
          Image3.png
          283 kB
        5. Image4.png
          Image4.png
          277 kB
        6. Image5.png
          Image5.png
          253 kB
        7. Image6.png
          Image6.png
          86 kB

          Issue Links

            Activity

            nfreese Nowlan Freese created issue -
            nfreese Nowlan Freese made changes -
            Field Original Value New Value
            Epic Link IGBF-1765 [ 17855 ]
            nfreese Nowlan Freese made changes -
            Link This issue relates to IGBF-1217 [ IGBF-1217 ]
            nfreese Nowlan Freese made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            nfreese Nowlan Freese added a comment -

            To test in IGB:

            Add this link as a new Quickload under the Data Access tab in Available Data - Configure:
            https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.1/quickload/

            Open the H_sapiens_Dec_2013 human genome.
            Navigate to chr1:80,644,218-80,644,359
            Select one of the Quickload files and click Load Data.
            Look to see that the data load and no errors are thrown.
            Repeat for every Quickload file.

            Open the H_sapiens_Feb_2009 human genome.
            Navigate to chr1:77,876,660-77,876,761
            Select one of the Quickload files and click Load Data.
            Look to see that the data load and no errors are thrown.
            Repeat for every Quickload file.

            Show
            nfreese Nowlan Freese added a comment - To test in IGB: Add this link as a new Quickload under the Data Access tab in Available Data - Configure: https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.1/quickload/ Open the H_sapiens_Dec_2013 human genome. Navigate to chr1:80,644,218-80,644,359 Select one of the Quickload files and click Load Data. Look to see that the data load and no errors are thrown. Repeat for every Quickload file. Open the H_sapiens_Feb_2009 human genome. Navigate to chr1:77,876,660-77,876,761 Select one of the Quickload files and click Load Data. Look to see that the data load and no errors are thrown. Repeat for every Quickload file.
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ]
            nfreese Nowlan Freese made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            ssegu Sai Supreeth Segu (Inactive) made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            ssegu Sai Supreeth Segu (Inactive) made changes -
            Assignee Sai Supreeth Segu [ ssegu ]
            Hide
            ssegu Sai Supreeth Segu (Inactive) added a comment - - edited

            I have tested each quickload file in the 2 scenarios.

            When I tested in the first scenario using H_sapiens_Dec_2013 human genome, part of the data was not loaded and mostly they are Soft-clips.
            I have added 5 images for that scenario and log for the error is:

            15:31:13.752 WARN o.l.i.quickload.util.QuickloadUtils - Optional species.txt could not be loaded from: https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.1/quickload/species.txt
            15:31:13.826 INFO o.l.i.appstore.IgbAppServerLauncher - Started REST endpoint.

            When I tested in the first scenario using H_sapiens_Dec_2009 human genome, this issue is not there but in AshkenazimTrio --> Son --> Illumina_6kb_matepair was not loaded and log for the error is:

            17:31:38.694 INFO c.a.igb.view.load.GeneralLoadUtils - Loaded RefGene in 2.188 s
            org.lorainelab.igb.bam.BAM$BamIndexNotFoundException: Could not find Bam Index File.
            at org.lorainelab.igb.bam.BAM.findIndexFile(BAM.java:433)
            at org.lorainelab.igb.bam.BAM.getBamIndexUriStr(BAM.java:139)
            at org.lorainelab.igb.bam.BAM.getSAMFileReader(BAM.java:98)
            at org.lorainelab.igb.bam.BAM.init(BAM.java:157)
            at org.lorainelab.igb.bam.XAM.getChromosomeList(XAM.java:100)
            at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadAndAddSymmetries(QuickLoadSymLoader.java:153)
            at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadSymmetriesThread(QuickLoadSymLoader.java:139)
            at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadFeatures(QuickLoadSymLoader.java:119)
            at com.affymetrix.igb.view.load.GeneralLoadUtils.loadFeaturesForSym(GeneralLoadUtils.java:751)
            at com.affymetrix.igb.view.load.GeneralLoadUtils$2.runInBackground(GeneralLoadUtils.java:693)
            at com.affymetrix.igb.view.load.GeneralLoadUtils$2.runInBackground(GeneralLoadUtils.java:688)
            at com.affymetrix.genometry.thread.CThreadWorker.doInBackground(CThreadWorker.java:73)
            at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at javax.swing.SwingWorker.run(SwingWorker.java:334)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)

            cc: Dr.Nowlan Freese

            Show
            ssegu Sai Supreeth Segu (Inactive) added a comment - - edited I have tested each quickload file in the 2 scenarios. When I tested in the first scenario using H_sapiens_Dec_2013 human genome, part of the data was not loaded and mostly they are Soft-clips. I have added 5 images for that scenario and log for the error is: 15:31:13.752 WARN o.l.i.quickload.util.QuickloadUtils - Optional species.txt could not be loaded from: https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.1/quickload/species.txt 15:31:13.826 INFO o.l.i.appstore.IgbAppServerLauncher - Started REST endpoint. When I tested in the first scenario using H_sapiens_Dec_2009 human genome, this issue is not there but in AshkenazimTrio --> Son --> Illumina_6kb_matepair was not loaded and log for the error is: 17:31:38.694 INFO c.a.igb.view.load.GeneralLoadUtils - Loaded RefGene in 2.188 s org.lorainelab.igb.bam.BAM$BamIndexNotFoundException: Could not find Bam Index File. at org.lorainelab.igb.bam.BAM.findIndexFile(BAM.java:433) at org.lorainelab.igb.bam.BAM.getBamIndexUriStr(BAM.java:139) at org.lorainelab.igb.bam.BAM.getSAMFileReader(BAM.java:98) at org.lorainelab.igb.bam.BAM.init(BAM.java:157) at org.lorainelab.igb.bam.XAM.getChromosomeList(XAM.java:100) at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadAndAddSymmetries(QuickLoadSymLoader.java:153) at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadSymmetriesThread(QuickLoadSymLoader.java:139) at com.affymetrix.genometry.quickload.QuickLoadSymLoader.loadFeatures(QuickLoadSymLoader.java:119) at com.affymetrix.igb.view.load.GeneralLoadUtils.loadFeaturesForSym(GeneralLoadUtils.java:751) at com.affymetrix.igb.view.load.GeneralLoadUtils$2.runInBackground(GeneralLoadUtils.java:693) at com.affymetrix.igb.view.load.GeneralLoadUtils$2.runInBackground(GeneralLoadUtils.java:688) at com.affymetrix.genometry.thread.CThreadWorker.doInBackground(CThreadWorker.java:73) at javax.swing.SwingWorker$1.call(SwingWorker.java:295) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at javax.swing.SwingWorker.run(SwingWorker.java:334) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) cc: Dr. Nowlan Freese
            ssegu Sai Supreeth Segu (Inactive) made changes -
            Attachment Image1.png [ 15146 ]
            Attachment Image2.png [ 15147 ]
            Attachment Image3.png [ 15148 ]
            Attachment Image4.png [ 15149 ]
            Attachment Image5.png [ 15150 ]
            Attachment Image6.png [ 15151 ]
            ssegu Sai Supreeth Segu (Inactive) made changes -
            Status First Level Review in Progress [ 10301 ] To-Do [ 10305 ]
            ssegu Sai Supreeth Segu (Inactive) made changes -
            Assignee Sai Supreeth Segu [ ssegu ]
            ann.loraine Ann Loraine made changes -
            Sprint Winter 3 Jan 25 - Feb 5 [ 113 ] Winter 3 Jan 25 - Feb 5, Winter 4 Feb 8 - Feb 19 [ 113, 114 ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ]
            nfreese Nowlan Freese made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            nfreese Nowlan Freese added a comment -

            I'm not seeing any issues with the 2013 Human genome from the log or images 1-5.

            The AshkenazimTrio --> Son --> Illumina_6kb_matepair was pointing at the wrong bam file. I have updated the annots.xml and it is now working correctly.

            Next step: I need to update igbDefaultPrefs.json

            Show
            nfreese Nowlan Freese added a comment - I'm not seeing any issues with the 2013 Human genome from the log or images 1-5. The AshkenazimTrio --> Son --> Illumina_6kb_matepair was pointing at the wrong bam file. I have updated the annots.xml and it is now working correctly. Next step: I need to update igbDefaultPrefs.json
            nfreese Nowlan Freese made changes -
            Attachment giab_s3_urls.zip [ 15170 ]
            nfreese Nowlan Freese made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            nfreese Nowlan Freese made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            nfreese Nowlan Freese made changes -
            Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ]
            Hide
            nfreese Nowlan Freese added a comment -

            A number of comments are included in this declined pull request.

            I think the least amount of work for us would be to leave igbDefaultPrefs.json pointed at the 1.0.0 branch. If we need to update a link in the annots.xml, the change will be made to the 1.0.0 branch, but no new branch will be created. This way all versions of IGB will have the most up to date Quickload info and there will still be a record of the changes (commit history for that branch). If at some point in the future we create a new Quickload format then we could create a new branch and subsequent versions of IGB would point at the new branch.

            Show
            nfreese Nowlan Freese added a comment - A number of comments are included in this declined pull request . I think the least amount of work for us would be to leave igbDefaultPrefs.json pointed at the 1.0.0 branch. If we need to update a link in the annots.xml, the change will be made to the 1.0.0 branch, but no new branch will be created. This way all versions of IGB will have the most up to date Quickload info and there will still be a record of the changes (commit history for that branch). If at some point in the future we create a new Quickload format then we could create a new branch and subsequent versions of IGB would point at the new branch.
            Hide
            nfreese Nowlan Freese added a comment -

            All of the changes have been merged into the release-v1.0.0 branch. No changes need to be made to IGB. I have tested that AshkenazimTrio --> Son --> Illumina_6kb_matepair works in IGB 9.1.8.

            Closing issue.

            Show
            nfreese Nowlan Freese added a comment - All of the changes have been merged into the release-v1.0.0 branch. No changes need to be made to IGB. I have tested that AshkenazimTrio --> Son --> Illumina_6kb_matepair works in IGB 9.1.8. Closing issue.
            nfreese Nowlan Freese made changes -
            Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
            nfreese Nowlan Freese made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            nfreese Nowlan Freese made changes -
            Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
            nfreese Nowlan Freese made changes -
            Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
            nfreese Nowlan Freese made changes -
            Resolution Done [ 10000 ]
            Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ]

              People

              • Assignee:
                nfreese Nowlan Freese
                Reporter:
                nfreese Nowlan Freese
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: