Details

      Description

      Situation: A user has asked for the UCSC DAS functionality to be returned to IGB.

      Task: Revert the changes to remove DASv1 support from IGB.

        Attachments

          Issue Links

            Activity

            nfreese Nowlan Freese created issue -
            nfreese Nowlan Freese made changes -
            Field Original Value New Value
            Epic Link IGBF-1765 [ 17855 ]
            nfreese Nowlan Freese made changes -
            Link This issue relates to HELP-370 [ HELP-370 ]
            nfreese Nowlan Freese made changes -
            Link This issue relates to IGBF-2474 [ IGBF-2474 ]
            nfreese Nowlan Freese made changes -
            Fix Version/s 9.1.8 Major Release [ 10605 ]
            nfreese Nowlan Freese made changes -
            Link This issue relates to IGBF-2777 [ IGBF-2777 ]
            nfreese Nowlan Freese made changes -
            Link This issue relates to IGBF-2776 [ IGBF-2776 ]
            nfreese Nowlan Freese made changes -
            Summary Add back DAS support Restore DASv1 support
            nfreese Nowlan Freese made changes -
            Description Situation: A user has asked for the UCSC DAS functionality to be returned to IGB.

            Task: Revert the changes to remove DAS support from IGB.
            Situation: A user has asked for the UCSC DAS functionality to be returned to IGB.

            Task: Revert the changes to remove DASv1 support from IGB.
            nfreese Nowlan Freese made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            nfreese Nowlan Freese added a comment -

            Branch: https://bitbucket.org/nfreese/nowlanfork-igb/branch/IGBF-2793

            To test:
            Open the mouse genome (M_musculus_Dec_2011)
            Navigate to chr1:52,182,882-52,235,630
            Select the External View tab
            In the dropdown menu, select UCSC (should be UCSC by default)
            Click update (Gene annotations should appear in the External View tab)
            Next, select View > View Region in UCSC Browser (a web browser should open to genome.ucsc.edu)
            Next, select the Data Access tab
            Under Available Data, there should be a folder for UCSC (DAS)
            Click the arrow to see all of the options under UCSC (DAS)
            Select ncbiRefSeqCurated (note that the DAS menu and data may move slowly as they utilize an external resource)
            Once the track has appeared in IGB, click Load Data (gene models should appear)

            Show
            nfreese Nowlan Freese added a comment - Branch: https://bitbucket.org/nfreese/nowlanfork-igb/branch/IGBF-2793 To test: Open the mouse genome (M_musculus_Dec_2011) Navigate to chr1:52,182,882-52,235,630 Select the External View tab In the dropdown menu, select UCSC (should be UCSC by default) Click update (Gene annotations should appear in the External View tab) Next, select View > View Region in UCSC Browser (a web browser should open to genome.ucsc.edu ) Next, select the Data Access tab Under Available Data, there should be a folder for UCSC (DAS) Click the arrow to see all of the options under UCSC (DAS) Select ncbiRefSeqCurated (note that the DAS menu and data may move slowly as they utilize an external resource) Once the track has appeared in IGB, click Load Data (gene models should appear)
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ]
            nfreese Nowlan Freese made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            Hide
            nfreese Nowlan Freese added a comment -

            Restoring DASv1 support in IGB fixes the issues with the External View tab (IGBF-2776) and the View Region in UCSC Browser (IGBF-2777), as they both used DAS to determine what genomes were available in UCSC (initUCSCSources() method in UCSCViewAction.java).

            The issue with the DAS files not loading in IGB was due to the XML being returned containing a link to a dead dtd file (http://www.biodas.org/dtd/dasgff.dtd). At some point in the last couple of years the biodas.org website appears to have gone offline. IGB versions prior to 8.4.0 did not validate the XML using dtd and thus had no issues. IGB versions 8.4.0 to 9.1.6 attempt to validate the XML using dtd and because the biodas.org link is no longer active failed to validate causing the DAS data to fail to load in IGB.

            Show
            nfreese Nowlan Freese added a comment - Restoring DASv1 support in IGB fixes the issues with the External View tab ( IGBF-2776 ) and the View Region in UCSC Browser ( IGBF-2777 ), as they both used DAS to determine what genomes were available in UCSC (initUCSCSources() method in UCSCViewAction.java). The issue with the DAS files not loading in IGB was due to the XML being returned containing a link to a dead dtd file ( http://www.biodas.org/dtd/dasgff.dtd ). At some point in the last couple of years the biodas.org website appears to have gone offline. IGB versions prior to 8.4.0 did not validate the XML using dtd and thus had no issues. IGB versions 8.4.0 to 9.1.6 attempt to validate the XML using dtd and because the biodas.org link is no longer active failed to validate causing the DAS data to fail to load in IGB.
            nfreese Nowlan Freese made changes -
            Link This issue relates to HELP-374 [ HELP-374 ]
            Hide
            nfreese Nowlan Freese added a comment - - edited

            Many files were removed/edited when DASv1 support was removed from IGB. I reverted three commits to restore the functionality, but this required resolving several merge conflicts. In addition, I tried to maintain the removal of DASv2 support as it does not appear to be necessary. However, many of the merge conflicts were due to commenting out both DASv1 and DASv2 at the same time.

            Show
            nfreese Nowlan Freese added a comment - - edited Many files were removed/edited when DASv1 support was removed from IGB. I reverted three commits to restore the functionality, but this required resolving several merge conflicts. In addition, I tried to maintain the removal of DASv2 support as it does not appear to be necessary. However, many of the merge conflicts were due to commenting out both DASv1 and DASv2 at the same time.
            Hide
            nfreese Nowlan Freese added a comment - - edited

            Many of the functions provided by DAS can be accomplished with the newer UCSC Rest API. At some point it would be good to replace the DAS functionality with the REST API, though this may require extensive work.

            Show
            nfreese Nowlan Freese added a comment - - edited Many of the functions provided by DAS can be accomplished with the newer UCSC Rest API . At some point it would be good to replace the DAS functionality with the REST API, though this may require extensive work.
            rweidenh Logan Weidenhammer (Inactive) made changes -
            Assignee Rachel Weidenhammer [ rweidenh ]
            rweidenh Logan Weidenhammer (Inactive) made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            Hide
            rweidenh Logan Weidenhammer (Inactive) added a comment - - edited

            (reviewed on Windows 10 OS)
            Functional Review:

            Following the steps from the first comment works as instructed.
            I encountered no errors.

            Code Review:

            I don't feel like I am able to understand the changes well enough to review them.
            It looks like the DAS Parser was told not to support DTD, and some other files were added but then removed.

            But thank you for all of the information in the previous comments on this issue. They are very informative and helpful.

            Show
            rweidenh Logan Weidenhammer (Inactive) added a comment - - edited (reviewed on Windows 10 OS) Functional Review: Following the steps from the first comment works as instructed. I encountered no errors. Code Review: I don't feel like I am able to understand the changes well enough to review them. It looks like the DAS Parser was told not to support DTD, and some other files were added but then removed. But thank you for all of the information in the previous comments on this issue. They are very informative and helpful.
            rweidenh Logan Weidenhammer (Inactive) made changes -
            Status First Level Review in Progress [ 10301 ] Needs 1st Level Review [ 10005 ]
            rweidenh Logan Weidenhammer (Inactive) made changes -
            Assignee Rachel Weidenhammer [ rweidenh ]
            rweidenh Logan Weidenhammer (Inactive) made changes -
            Assignee Nowlan Freese [ nfreese ]
            rweidenh Logan Weidenhammer (Inactive) made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            rweidenh Logan Weidenhammer (Inactive) made changes -
            Status First Level Review in Progress [ 10301 ] To-Do [ 10305 ]
            nfreese Nowlan Freese made changes -
            Status To-Do [ 10305 ] Pull Request Submitted [ 10101 ]
            Show
            nfreese Nowlan Freese added a comment - Pull request: https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/865/igbf-2793
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ]
            ann.loraine Ann Loraine made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            ann.loraine Ann Loraine made changes -
            Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Assignee Philip Badzuh [ pbadzuh ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
            Hide
            pbadzuh Philip Badzuh (Inactive) added a comment -

            Testing on macOS following the instructions in the first comment, everything seems to be working as expected. Closing issue.

            Show
            pbadzuh Philip Badzuh (Inactive) added a comment - Testing on macOS following the instructions in the first comment, everything seems to be working as expected. Closing issue.
            pbadzuh Philip Badzuh (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Assignee Philip Badzuh [ pbadzuh ] Nowlan Freese [ nfreese ]
            ann.loraine Ann Loraine made changes -
            Resolution Done [ 10000 ]
            Status Closed [ 6 ] To-Do [ 10305 ]
            Hide
            ann.loraine Ann Loraine added a comment -

            From the Wayback Machine.

            Show
            ann.loraine Ann Loraine added a comment - From the Wayback Machine.
            ann.loraine Ann Loraine made changes -
            Attachment dasgff.dtd [ 15492 ]
            nfreese Nowlan Freese made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            nfreese Nowlan Freese made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            nfreese Nowlan Freese made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            nfreese Nowlan Freese made changes -
            Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
            nfreese Nowlan Freese made changes -
            Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
            nfreese Nowlan Freese made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            nfreese Nowlan Freese made changes -
            Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
            nfreese Nowlan Freese made changes -
            Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
            nfreese Nowlan Freese made changes -
            Resolution Done [ 10000 ]
            Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]
            Hide
            nfreese Nowlan Freese added a comment - - edited

            I reopened this ticket as there was an issue with loading the sequence from genomes available through DAS.

            The issue is that the dtd file - http://www.biodas.org/dtd/dasdna.dtd - is no longer available. I could not find the file online including on the Wayback Machine.

            I have changed the IGB logic to ignore the dtd file when loading residues following logic from here and here.

            Branch: https://bitbucket.org/nfreese/nowlanfork-igb/branch/IGBF-2793

            If we are able to acquire the dtd file, it may be worth implementing a different solution where we use the file internally along with an entity resolver as discussed here and here.

            However, the best solution may be to begin using the UCSC API instead of DAS.

            Show
            nfreese Nowlan Freese added a comment - - edited I reopened this ticket as there was an issue with loading the sequence from genomes available through DAS. The issue is that the dtd file - http://www.biodas.org/dtd/dasdna.dtd - is no longer available. I could not find the file online including on the Wayback Machine. I have changed the IGB logic to ignore the dtd file when loading residues following logic from here and here . Branch: https://bitbucket.org/nfreese/nowlanfork-igb/branch/IGBF-2793 If we are able to acquire the dtd file, it may be worth implementing a different solution where we use the file internally along with an entity resolver as discussed here and here . However, the best solution may be to begin using the UCSC API instead of DAS.
            nfreese Nowlan Freese made changes -
            Resolution Done [ 10000 ]
            Status Closed [ 6 ] To-Do [ 10305 ]
            nfreese Nowlan Freese made changes -
            Sprint Winter 5 Feb 22 - Mar 5 [ 115 ] Winter 5 Feb 22 - Mar 5, Spring 3 2021 Apr 19 - Apr 30 [ 115, 119 ]
            nfreese Nowlan Freese made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            nfreese Nowlan Freese added a comment - - edited

            To test, download IGBF-2793 installer.

            In IGB with UCSC DAS enabled:
            Select the Caenorhabditis brenneri in the Species dropdown
            Select the C_brenneri_Feb_2008 Genome Version
            Navigate to Un:55,401,094-55,401,137
            Click Load Sequence

            The sequence should appear if working correctly.

            Show
            nfreese Nowlan Freese added a comment - - edited To test, download IGBF-2793 installer . In IGB with UCSC DAS enabled: Select the Caenorhabditis brenneri in the Species dropdown Select the C_brenneri_Feb_2008 Genome Version Navigate to Un:55,401,094-55,401,137 Click Load Sequence The sequence should appear if working correctly.
            nfreese Nowlan Freese made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            omarne Omkar Marne (Inactive) made changes -
            Assignee Nowlan Freese [ nfreese ] Omkar Marne [ omarne ]
            Hide
            omarne Omkar Marne (Inactive) added a comment -

            The sequence appeared as expected.

            Show
            omarne Omkar Marne (Inactive) added a comment - The sequence appeared as expected.
            omarne Omkar Marne (Inactive) made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            omarne Omkar Marne (Inactive) made changes -
            Assignee Omkar Marne [ omarne ] Nowlan Freese [ nfreese ]
            nfreese Nowlan Freese made changes -
            Story Points 2 4
            Show
            nfreese Nowlan Freese added a comment - Pull request: https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/876/igbf-2793-fix-das-load-sequence
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ]
            nfreese Nowlan Freese made changes -
            Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
            nfreese Nowlan Freese made changes -
            Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
            ann.loraine Ann Loraine made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            ann.loraine Ann Loraine made changes -
            Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
            Hide
            ann.loraine Ann Loraine added a comment -

            Merged to master branch and built master branch installers. Ready for testing.

            Show
            ann.loraine Ann Loraine added a comment - Merged to master branch and built master branch installers. Ready for testing.
            nfreese Nowlan Freese made changes -
            Assignee Rachel Weidenhammer [ rweidenh ]
            rweidenh Logan Weidenhammer (Inactive) made changes -
            Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
            Hide
            rweidenh Logan Weidenhammer (Inactive) added a comment -

            Tested on Windows.
            sequence loads.
            UCSC (DAS) appears in the Data Access panel and those files load as well.

            moving to closed.

            Show
            rweidenh Logan Weidenhammer (Inactive) added a comment - Tested on Windows. sequence loads. UCSC (DAS) appears in the Data Access panel and those files load as well. moving to closed.
            rweidenh Logan Weidenhammer (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]
            rweidenh Logan Weidenhammer (Inactive) made changes -
            Assignee Rachel Weidenhammer [ rweidenh ] Nowlan Freese [ nfreese ]
            ann.loraine Ann Loraine made changes -
            Link This issue relates to IGBF-2855 [ IGBF-2855 ]
            nfreese Nowlan Freese made changes -
            Link This issue relates to IGBF-781 [ IGBF-781 ]
            nfreese Nowlan Freese made changes -
            Link This issue relates to IGBF-2869 [ IGBF-2869 ]

              People

              • Assignee:
                nfreese Nowlan Freese
                Reporter:
                nfreese Nowlan Freese
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: