Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-2953

Sanitize annots.xml trackhub data source file URLs for use in IGB 9.1.8

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      Modify the annots.xml endpoint of the facade API to strip off any query parameters that may be at the end of a data file URL. This is required to allow IGB 9.1.8 to effectively determine the filetype when loading data files.

        Attachments

          Issue Links

            Activity

            pbadzuh Philip Badzuh (Inactive) created issue -
            pbadzuh Philip Badzuh (Inactive) made changes -
            Field Original Value New Value
            Epic Link IGBF-2831 [ 19524 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Link This issue relates to IGBF-2944 [ IGBF-2944 ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            ann.loraine Ann Loraine made changes -
            Sprint Fall 4 2021 Sep 27 - Oct 8 [ 130 ]
            ann.loraine Ann Loraine made changes -
            Sprint Spring 4 2022 Feb 14 - Feb 25 [ 139 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Assignee Philip Badzuh [ pbadzuh ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            pbadzuh Philip Badzuh (Inactive) added a comment -

            Please see my changes here.

            Post-deployment Testing:

            Show
            pbadzuh Philip Badzuh (Inactive) added a comment - Please see my changes here . Post-deployment Testing: Perform a GET request using curl or postman for the URL below https://translate.bioviz.org/api/?hubUrl=https://api.wenglab.org/screen_v13/hubs/rna22/hub.txt&filePath=/hg38/annots.xml Ensure that none of the file elements have name attributes with query parameters.
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Assignee Philip Badzuh [ pbadzuh ]
            ann.loraine Ann Loraine made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ]
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            What happens when igb tries to retrieve the data? Will the server send the data or generate an error?

            If the query parameters are necessary to get the requested data, then I think it will probably be necessary to create some kind of proxy server or redirection server on the trackhub facade host site.

            How this might work (a suggestion):

            • The remote host for the data file expects a URL such as example.com/a_very_weird_path
            • The track hub facade realizes this and understands that the format of the file needs to be indicated in a file extension, e.g., "something.bed", in order for IGB to properly request and parse the data (in this case, the format is "BED")
            • The track hub therefore provides a different URL to IGB, e.g., translate.bioviz.org/a_very_weird_path/something.bed
            • When IGB asks for translate.bioviz.org/a_very_weird_path/something.bed, probably as a byte range HTTP request, the facade host retrieves those data from the remote source using example.com/a_very_weird_path. Like a proxy server, the facade host simply "copies" the HTTP request coming from IGB, substituting the proper URL
            Show
            ann.loraine Ann Loraine added a comment - - edited What happens when igb tries to retrieve the data? Will the server send the data or generate an error? If the query parameters are necessary to get the requested data, then I think it will probably be necessary to create some kind of proxy server or redirection server on the trackhub facade host site. How this might work (a suggestion): The remote host for the data file expects a URL such as example.com/a_very_weird_path The track hub facade realizes this and understands that the format of the file needs to be indicated in a file extension, e.g., "something.bed", in order for IGB to properly request and parse the data (in this case, the format is "BED") The track hub therefore provides a different URL to IGB, e.g., translate.bioviz.org/a_very_weird_path/something.bed When IGB asks for translate.bioviz.org/a_very_weird_path/something.bed, probably as a byte range HTTP request, the facade host retrieves those data from the remote source using example.com/a_very_weird_path. Like a proxy server, the facade host simply "copies" the HTTP request coming from IGB, substituting the proper URL
            Hide
            ann.loraine Ann Loraine added a comment -

            Note that the above suggestion is a big change for how we do things, because this could potentially pull a massive amount of data through the facade host, depending on the requests IGB is making. There might be a better way, in which the facade host would instead use a simple redirection to forward IGB's request to the host server, after changing the URL. If this is possible, then the remote host will send the data directly to IGB, without the data having to flow through facade host, which would be wasteful.

            Show
            ann.loraine Ann Loraine added a comment - Note that the above suggestion is a big change for how we do things, because this could potentially pull a massive amount of data through the facade host, depending on the requests IGB is making. There might be a better way, in which the facade host would instead use a simple redirection to forward IGB's request to the host server, after changing the URL. If this is possible, then the remote host will send the data directly to IGB, without the data having to flow through facade host, which would be wasteful.
            ann.loraine Ann Loraine made changes -
            Status First Level Review in Progress [ 10301 ] To-Do [ 10305 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ] Philip Badzuh [ pbadzuh ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            pbadzuh Philip Badzuh (Inactive) added a comment - - edited

            I agree that forwarding the request would be better, although to do so, we would need to store a mapping of all data urls that contain query params to their stripped counterparts. This would probably be best achieved by using some sort of database. It might be easier to just update the IGB code to correctly handle these URLs, although I'm not sure how easy this would be. [~aloraine], what do you think?

            Show
            pbadzuh Philip Badzuh (Inactive) added a comment - - edited I agree that forwarding the request would be better, although to do so, we would need to store a mapping of all data urls that contain query params to their stripped counterparts. This would probably be best achieved by using some sort of database. It might be easier to just update the IGB code to correctly handle these URLs, although I'm not sure how easy this would be. [~aloraine] , what do you think?
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Assignee Philip Badzuh [ pbadzuh ]
            ann.loraine Ann Loraine made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            Hide
            ann.loraine Ann Loraine added a comment -

            We will release a new version of IGB that addresses the issue. In light of this, we will hold off on changing the back end code for now.

            Show
            ann.loraine Ann Loraine added a comment - We will release a new version of IGB that addresses the issue. In light of this, we will hold off on changing the back end code for now.
            ann.loraine Ann Loraine made changes -
            Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
            ann.loraine Ann Loraine made changes -
            Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
            ann.loraine Ann Loraine made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            ann.loraine Ann Loraine made changes -
            Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
            ann.loraine Ann Loraine made changes -
            Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
            ann.loraine Ann Loraine made changes -
            Resolution Done [ 10000 ]
            Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]
            ann.loraine Ann Loraine made changes -
            Assignee Philip Badzuh [ pbadzuh ]

              People

              • Assignee:
                pbadzuh Philip Badzuh (Inactive)
                Reporter:
                pbadzuh Philip Badzuh (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: