Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-2898

Add listing of UCSC hubs to the bioviz trackhubs page

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Story Points:
      3
    • Sprint:
      Summer 3 2021 Jul 12 - Jul 23, Summer 4 2021 Aug 2 - Aug 13, Fall 1 2021 Aug 16 - Aug 27

      Description

      Scrape available UCSC track hub URLs and allow users of the bioviz trackhub UI to:

      • scroll through the list of available URLs
      • copy URLs and/or populate the converter input field
      • search URLs by the text contained within

      A collection of public hubs can be found here.

        Attachments

          Issue Links

            Activity

            pbadzuh Philip Badzuh (Inactive) created issue -
            pbadzuh Philip Badzuh (Inactive) made changes -
            Field Original Value New Value
            Epic Link IGBF-1765 [ 17855 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Link This issue blocks IGBF-2892 [ IGBF-2892 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Epic Link IGBF-1765 [ 17855 ] IGBF-2831 [ 19524 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status In Progress [ 3 ] To-Do [ 10305 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Description Scrape available UCSC track hub URLs and allow users of the bioviz trackhub UI to:
            - scroll through the list of available URLs
            - copy URLs and/or populate the converter input field
            - search URLs by the text contained within
            Scrape available UCSC track hub URLs and allow users of the bioviz trackhub UI to:
            - scroll through the list of available URLs
            - copy URLs and/or populate the converter input field
            - search URLs by the text contained within

            Collection of public hubs can be found [here|https://genome.ucsc.edu/cgi-bin/hgHubConnect]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Description Scrape available UCSC track hub URLs and allow users of the bioviz trackhub UI to:
            - scroll through the list of available URLs
            - copy URLs and/or populate the converter input field
            - search URLs by the text contained within

            Collection of public hubs can be found [here|https://genome.ucsc.edu/cgi-bin/hgHubConnect]
            Scrape available UCSC track hub URLs and allow users of the bioviz trackhub UI to:
            - scroll through the list of available URLs
            - copy URLs and/or populate the converter input field
            - search URLs by the text contained within

            A collection of public hubs can be found [here|https://genome.ucsc.edu/cgi-bin/hgHubConnect#publicHubs]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Description Scrape available UCSC track hub URLs and allow users of the bioviz trackhub UI to:
            - scroll through the list of available URLs
            - copy URLs and/or populate the converter input field
            - search URLs by the text contained within

            A collection of public hubs can be found [here|https://genome.ucsc.edu/cgi-bin/hgHubConnect#publicHubs]
            Scrape available UCSC track hub URLs and allow users of the bioviz trackhub UI to:
            - scroll through the list of available URLs
            - copy URLs and/or populate the converter input field
            - search URLs by the text contained within

            A collection of public hubs can be found [here|https://genome.ucsc.edu/cgi-bin/hgHubConnect#publicHubs].
            ann.loraine Ann Loraine made changes -
            Description Scrape available UCSC track hub URLs and allow users of the bioviz trackhub UI to:
            - scroll through the list of available URLs
            - copy URLs and/or populate the converter input field
            - search URLs by the text contained within

            A collection of public hubs can be found [here|https://genome.ucsc.edu/cgi-bin/hgHubConnect#publicHubs].
            Scrape available UCSC track hub URLs and allow users of the bioviz trackhub UI to:
            - scroll through the list of available URLs
            - copy URLs and/or populate the converter input field
            - search URLs by the text contained within

            A collection of public hubs can be found [here|https://genome.ucsc.edu/cgi-bin/hgHubConnect#publicHubs].

            Hide
            ann.loraine Ann Loraine added a comment -

            To get the track hub address from the above link:

            • Click on an item in column labeled "Hub Name"
            • Copy the URL shown in the browser

            This URL is the "track hub URL".

            Show
            ann.loraine Ann Loraine added a comment - To get the track hub address from the above link: Click on an item in column labeled "Hub Name" Copy the URL shown in the browser This URL is the "track hub URL".
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            nfreese Nowlan Freese added a comment -

            Philip found this API for retrieving the public hubs: https://api.genome.ucsc.edu/list/publicHubs

            Show
            nfreese Nowlan Freese added a comment - Philip found this API for retrieving the public hubs: https://api.genome.ucsc.edu/list/publicHubs
            pbadzuh Philip Badzuh (Inactive) made changes -
            Link This issue relates to IGBF-2903 [ IGBF-2903 ]
            Hide
            pbadzuh Philip Badzuh (Inactive) added a comment - - edited

            Please see my current implementation deployed here. Any feedback or suggestions are welcome.

            The respective code can be found here.

            Show
            pbadzuh Philip Badzuh (Inactive) added a comment - - edited Please see my current implementation deployed here . Any feedback or suggestions are welcome. The respective code can be found here .
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Assignee Philip Badzuh [ pbadzuh ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Story Points 2 3
            ann.loraine Ann Loraine made changes -
            Sprint Summer 3 2021 Jul 12 - Jul 23 [ 125 ] Summer 3 2021 Jul 12 - Jul 23, Summer 4 2021 Aug 2 - Aug 13 [ 125, 126 ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            ann.loraine Ann Loraine made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ]
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Redeploying branch IGBF-2898 from repository https://bitbucket.org/pbadzuh/bioviz_pbdev.git to https://bioviztest.bioviz.org using new playbook developed in IGBF-2904 "Configure playbooks to specify deployed facade site for bioviz".

            Using inventory file:

            [targets]
            bioviztest ansible_host=52.4.208.199 domain=bioviztest.bioviz.org stack=Develop  igb_branch=release-9.1.8 igb_version=9.1.8 igb_repository=https://bitbucket.org/lorainelab/integrated-genome-browser bioviz_repo=https://bitbucket.org/pbadzuh/bioviz_pbdev.git bioviz_version=IGBF-2898
            
            [targets:vars]
            ansible_ssh_common_args="-o StrictHostKeyChecking=no"
            ansible_python_interpreter=/usr/bin/python
            ansible_ssh_user=ec2-user
            
            Show
            ann.loraine Ann Loraine added a comment - - edited Redeploying branch IGBF-2898 from repository https://bitbucket.org/pbadzuh/bioviz_pbdev.git to https://bioviztest.bioviz.org using new playbook developed in IGBF-2904 "Configure playbooks to specify deployed facade site for bioviz". Using inventory file: [targets] bioviztest ansible_host=52.4.208.199 domain=bioviztest.bioviz.org stack=Develop igb_branch=release-9.1.8 igb_version=9.1.8 igb_repository=https: //bitbucket.org/lorainelab/integrated-genome-browser bioviz_repo=https://bitbucket.org/pbadzuh/bioviz_pbdev.git bioviz_version=IGBF-2898 [targets:vars] ansible_ssh_common_args= "-o StrictHostKeyChecking=no" ansible_python_interpreter=/usr/bin/python ansible_ssh_user=ec2-user
            Hide
            ann.loraine Ann Loraine added a comment -

            See attached edit screen capture image (file name 2021-08-10-Suggestions.png) for suggested improvements and possible bug.

            Show
            ann.loraine Ann Loraine added a comment - See attached edit screen capture image (file name 2021-08-10-Suggestions.png) for suggested improvements and possible bug.
            ann.loraine Ann Loraine made changes -
            Attachment 2021-08-10-Suggestions.png [ 16778 ]
            ann.loraine Ann Loraine made changes -
            Status First Level Review in Progress [ 10301 ] To-Do [ 10305 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ] Philip Badzuh [ pbadzuh ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            ann.loraine Ann Loraine added a comment -

            Update: The "copy" and "refresh" icons did not work for me at first, but then I did a "hard reload" of the page, and now they do work. So I'm now able to use the new version of the page to generate new Quickload data sources.

            Additional feature requests:

            • Can you allow searching by genome version name? If I am looking at my own data and would like to compare it to data sets from other groups, it would be helpful to be able to search using the name of the reference genome I'm working with, e.g., "hg38", the synonym for IGB genome version name H_sapiens_Dec_2013.
            • Can you modify the interface so that the section of the page titled "Next, add the URL below as a new data source in IGB" always appears immediately below the "Convert" button?
            • Add a parenthetical after each species name indicating the genome version (or versions) available for that species in each Track Hub. For example: "Home sapiens (hg38)"
            • Just a reminder: the example link needs to be updated. Also, after the example link, we should probably let people know which genome they should open in IGB. The example used currently refers to the older version of the human genome. Maybe we ought to pick a more up-to-date one, something that would be interesting enough to serve as a good demonstration.
            Show
            ann.loraine Ann Loraine added a comment - Update: The "copy" and "refresh" icons did not work for me at first, but then I did a "hard reload" of the page, and now they do work. So I'm now able to use the new version of the page to generate new Quickload data sources. Additional feature requests: Can you allow searching by genome version name? If I am looking at my own data and would like to compare it to data sets from other groups, it would be helpful to be able to search using the name of the reference genome I'm working with, e.g., "hg38", the synonym for IGB genome version name H_sapiens_Dec_2013. Can you modify the interface so that the section of the page titled "Next, add the URL below as a new data source in IGB" always appears immediately below the "Convert" button? Add a parenthetical after each species name indicating the genome version (or versions) available for that species in each Track Hub. For example: "Home sapiens (hg38)" Just a reminder: the example link needs to be updated. Also, after the example link, we should probably let people know which genome they should open in IGB. The example used currently refers to the older version of the human genome. Maybe we ought to pick a more up-to-date one, something that would be interesting enough to serve as a good demonstration.
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Just wanted to make a comment: The table is really great. It makes the software 10 times more useful!

            Currently I am using the new table to look for RNA-Seq or transcriptome data sets I can use to investigate whether or not the homeobox-minus MEOX1 splice variant, which is lowly expressed in healthy human tissues, is more highly expressed in cancerous tissues, a follow-up from the use case scenario in this IGB slidedeck.

            Show
            ann.loraine Ann Loraine added a comment - - edited Just wanted to make a comment: The table is really great. It makes the software 10 times more useful! Currently I am using the new table to look for RNA-Seq or transcriptome data sets I can use to investigate whether or not the homeobox-minus MEOX1 splice variant, which is lowly expressed in healthy human tissues, is more highly expressed in cancerous tissues, a follow-up from the use case scenario in this IGB slidedeck .
            Hide
            ann.loraine Ann Loraine added a comment -

            Request:

            • Prepend the protocol to the copied text so that users can copy and paste it into the IGB window without having to type of protocol string manually.
            Show
            ann.loraine Ann Loraine added a comment - Request: Prepend the protocol to the copied text so that users can copy and paste it into the IGB window without having to type of protocol string manually.
            ann.loraine Ann Loraine made changes -
            Sprint Summer 3 2021 Jul 12 - Jul 23, Summer 4 2021 Aug 2 - Aug 13 [ 125, 126 ] Summer 3 2021 Jul 12 - Jul 23, Summer 4 2021 Aug 2 - Aug 13, Fall 1 2021 Aug 16 - Aug 27 [ 125, 126, 127 ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            Hide
            ann.loraine Ann Loraine added a comment -

            For today's walkthru, deployed latest IGBF-2898 branch to bioviztest.bioviz.org and latest main branch to TestHub1.bioviz.org.

            attn: Philip Badzuh

            Show
            ann.loraine Ann Loraine added a comment - For today's walkthru, deployed latest IGBF-2898 branch to bioviztest.bioviz.org and latest main branch to TestHub1.bioviz.org. attn: Philip Badzuh
            Hide
            ann.loraine Ann Loraine added a comment -

            Error:

            When I click the "copy" arrow under the text entry box to try out the example track hub, the interface reports and error: "Please provide a valid track hub URL. See the example below for more details."

            Two problems:

            • The example does not work
            • The error message says "See the example below for more details" but there are no details or help provided.
            Show
            ann.loraine Ann Loraine added a comment - Error: When I click the "copy" arrow under the text entry box to try out the example track hub, the interface reports and error: "Please provide a valid track hub URL. See the example below for more details." Two problems: The example does not work The error message says "See the example below for more details" but there are no details or help provided.
            Hide
            ann.loraine Ann Loraine added a comment -

            Testing branch IGBF-2898 from https://bitbucket.org/pbadzuh/bioviz_pbdev.git, commit:

            decdf61 (HEAD -> IGBF-2898, origin/IGBF-2898) IGBF-2892: Cache public hub data beyond single user session
            

            The table reports "Unknown" followed by genome name in every cell, in the column "Organisms and Genomes".

            Do this for the next iteration:

            • Change the column heading to "Genome Versions"
            • Do not report "Unknown" for anything. If there is nothing to report, print nothing.
            • If you can identify the organism or a given genome, print its species name in parentheses, e.g., hg19 (Homo sapiens). If not, print the name of the genome version.
            Show
            ann.loraine Ann Loraine added a comment - Testing branch IGBF-2898 from https://bitbucket.org/pbadzuh/bioviz_pbdev.git , commit: decdf61 (HEAD -> IGBF-2898, origin/IGBF-2898) IGBF-2892: Cache public hub data beyond single user session The table reports "Unknown" followed by genome name in every cell, in the column "Organisms and Genomes". Do this for the next iteration: Change the column heading to "Genome Versions" Do not report "Unknown" for anything. If there is nothing to report, print nothing. If you can identify the organism or a given genome, print its species name in parentheses, e.g., hg19 (Homo sapiens). If not, print the name of the genome version.
            Hide
            ann.loraine Ann Loraine added a comment -

            Usability improvements - let's try these:

            • Add a column that numbers the rows - this should be column 1. Scientists (our users) are quantitatively minded. They will want to be able to know how many track hubs contain data for the genome version they are using in their research, or the number of rows returned from a search. This will also help us test the interface for correctness.
            • Move "actions" column to the front of the table, after the numbering column. As users learn the interface and the available Track Hubs, they will barely look at the rest of the information.
            Show
            ann.loraine Ann Loraine added a comment - Usability improvements - let's try these: Add a column that numbers the rows - this should be column 1. Scientists (our users) are quantitatively minded. They will want to be able to know how many track hubs contain data for the genome version they are using in their research, or the number of rows returned from a search. This will also help us test the interface for correctness. Move "actions" column to the front of the table, after the numbering column. As users learn the interface and the available Track Hubs, they will barely look at the rest of the information.
            Hide
            pbadzuh Philip Badzuh (Inactive) added a comment - - edited

            Merging existing changes to separate the below tasks into individual tickets.

            TODO:

            • fix url validation
            • open links in new tab
            • make tooltip and heading text more clear
            • change genome column's name and text content
            • make tooltip duration longer
            • move actions to table left - add fill arrow icon
            • move result section above table
            • ensure result section is in view during/after conversion
            • add info to result URL section
            • number rows
            • prevent result URL wrapping on hyphen

            Backend:

            • modify synonyms endpoint to ensure proper genome name in IGB header/topbar
            • add hub validation endpoint
            Show
            pbadzuh Philip Badzuh (Inactive) added a comment - - edited Merging existing changes to separate the below tasks into individual tickets. TODO: fix url validation open links in new tab make tooltip and heading text more clear change genome column's name and text content make tooltip duration longer move actions to table left - add fill arrow icon move result section above table ensure result section is in view during/after conversion add info to result URL section number rows prevent result URL wrapping on hyphen Backend: modify synonyms endpoint to ensure proper genome name in IGB header/topbar add hub validation endpoint
            Hide
            pbadzuh Philip Badzuh (Inactive) added a comment -

            Please see PR here.

            Show
            pbadzuh Philip Badzuh (Inactive) added a comment - Please see PR here .
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Assignee Philip Badzuh [ pbadzuh ]
            Hide
            ann.loraine Ann Loraine added a comment -

            Thank you Philip Badzuh for the TO-DO and Backend lists in the next-to-last comment before this one. Next step is to make individual JIRA issues for each one, using the bullet point text as titles, with maybe one or two minor edits, e.g., "Number the rows"

            Show
            ann.loraine Ann Loraine added a comment - Thank you Philip Badzuh for the TO-DO and Backend lists in the next-to-last comment before this one. Next step is to make individual JIRA issues for each one, using the bullet point text as titles, with maybe one or two minor edits, e.g., "Number the rows"
            ann.loraine Ann Loraine made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Link This issue relates to IGBF-2929 [ IGBF-2929 ]
            ann.loraine Ann Loraine made changes -
            Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
            Hide
            ann.loraine Ann Loraine added a comment -

            Merged and deployed for testing.

            Show
            ann.loraine Ann Loraine added a comment - Merged and deployed for testing.
            Hide
            ann.loraine Ann Loraine added a comment -

            Tested that the code properly deployed by reloading the page and observed that the "Track hub" link in the upper menu disappeared. Moving to closed.

            Show
            ann.loraine Ann Loraine added a comment - Tested that the code properly deployed by reloading the page and observed that the "Track hub" link in the upper menu disappeared. Moving to closed.
            ann.loraine Ann Loraine made changes -
            Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
            ann.loraine Ann Loraine made changes -
            Resolution Done [ 10000 ]
            Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ] Philip Badzuh [ pbadzuh ]

              People

              • Assignee:
                pbadzuh Philip Badzuh (Inactive)
                Reporter:
                pbadzuh Philip Badzuh (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: