Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Story Points:
      3
    • Sprint:
      Summer 2019 Sprint 11, Summer 2019 Sprint 12

      Description

      Task: Investigate how we could automate quickloads, how quickloads currently format data via annots.xml, where can we store quickloads or the data (google drive, dropbox, cyverse).

        Attachments

          Activity

          Hide
          nfreese Nowlan Freese added a comment -

          There is a python package called twobitreader for accessing twobit files, and includes a method for getting the chromosomes and their sizes.

          Show
          nfreese Nowlan Freese added a comment - There is a python package called twobitreader for accessing twobit files, and includes a method for getting the chromosomes and their sizes.
          Hide
          nfreese Nowlan Freese added a comment -

          The twoBitInfo source code can be downloaded from the UCSC store (free for academic users) as part of the Genome Browser source code.

          Show
          nfreese Nowlan Freese added a comment - The twoBitInfo source code can be downloaded from the UCSC store (free for academic users) as part of the Genome Browser source code.
          Hide
          ann.loraine Ann Loraine added a comment -

          A thought:
          Our server could generate the meta-data files "on the fly" as REST endpoints.

          Show
          ann.loraine Ann Loraine added a comment - A thought: Our server could generate the meta-data files "on the fly" as REST endpoints.
          Hide
          stiwari8 Srishti Tiwari (Inactive) added a comment -

          https://stackoverflow.com/questions/18116152/how-do-i-get-a-file-list-for-a-google-drive-public-hosted-folder
          This link seems to be helpful as it talks about accessing children list of files/folders of a public google drive folder link.

          Show
          stiwari8 Srishti Tiwari (Inactive) added a comment - https://stackoverflow.com/questions/18116152/how-do-i-get-a-file-list-for-a-google-drive-public-hosted-folder This link seems to be helpful as it talks about accessing children list of files/folders of a public google drive folder link.
          Hide
          stiwari8 Srishti Tiwari (Inactive) added a comment - - edited

          1. We plan to host quickload host website (say quickload.bioviz.org) which will require authentication from a service.
          2. For each user we intend to create a folder under which their multiple quickload related files will be stored. (annots.xml, contents.txt etc)
          3. We plan to store all genome.txts in one place and create symbolic links to each of those files in quickload sites folders referring to that genome.
          Refer image attached.

          Show
          stiwari8 Srishti Tiwari (Inactive) added a comment - - edited 1. We plan to host quickload host website (say quickload.bioviz.org) which will require authentication from a service. 2. For each user we intend to create a folder under which their multiple quickload related files will be stored. (annots.xml, contents.txt etc) 3. We plan to store all genome.txts in one place and create symbolic links to each of those files in quickload sites folders referring to that genome. Refer image attached.
          Hide
          nfreese Nowlan Freese added a comment -

          Need to make sure to include the index attribute . For bam files, IGB will assume the index file is available in the same directory, but since we will be linking to files, we will need to specify the link to the index itself.

          If the file is a BAM or tabix indexed file, then IGB assumes the index file has the same name as the target file with a standard extension appended. BAM file indexes have extension ".bai" and tabix indexes have extension ".tbi." However, if the index is in a different location or has a non-standard name, you can specify its location and file name using the index attribute.
          ex)
          index="http://www.example.com/indexfile.bai"

          Show
          nfreese Nowlan Freese added a comment - Need to make sure to include the index attribute . For bam files, IGB will assume the index file is available in the same directory, but since we will be linking to files, we will need to specify the link to the index itself. If the file is a BAM or tabix indexed file, then IGB assumes the index file has the same name as the target file with a standard extension appended. BAM file indexes have extension ".bai" and tabix indexes have extension ".tbi." However, if the index is in a different location or has a non-standard name, you can specify its location and file name using the index attribute. ex) index="http://www.example.com/indexfile.bai"
          Hide
          nfreese Nowlan Freese added a comment -

          Can control how the quickload directory structure appears in IGB by altering the "title" field.
          For example:

          title="Google/google_annotation.bed"

          will appear under the available data tab as a directory named Google containing the file google_annotation.bed. Note that google_annotation.bed does not need to be the actual name of the file, which is set in the "name" field.

          Show
          nfreese Nowlan Freese added a comment - Can control how the quickload directory structure appears in IGB by altering the "title" field. For example: title="Google/google_annotation.bed" will appear under the available data tab as a directory named Google containing the file google_annotation.bed. Note that google_annotation.bed does not need to be the actual name of the file, which is set in the "name" field.
          Hide
          nfreese Nowlan Freese added a comment - - edited

          Escaping & in the XML works.

          Show
          nfreese Nowlan Freese added a comment - - edited Escaping & in the XML works.
          Hide
          stiwari8 Srishti Tiwari (Inactive) added a comment - - edited

          Adding an & in the URL somehow makes the annots.xml invalid. We can workaround with it by replacing & in the URL with & and then it works fine.

          However, it only works for Linux and MAC. Currently, it is throwing exceptions in Windows IGB.

          Show
          stiwari8 Srishti Tiwari (Inactive) added a comment - - edited Adding an & in the URL somehow makes the annots.xml invalid. We can workaround with it by replacing & in the URL with & and then it works fine. However, it only works for Linux and MAC. Currently, it is throwing exceptions in Windows IGB.
          Hide
          nfreese Nowlan Freese added a comment -

          A Quickload can be hosted on CyVerse and will work correctly in IGB: https://data.cyverse.org/dav-anon/iplant/home/nowlanf/quickload/

          Show
          nfreese Nowlan Freese added a comment - A Quickload can be hosted on CyVerse and will work correctly in IGB: https://data.cyverse.org/dav-anon/iplant/home/nowlanf/quickload/
          Hide
          nfreese Nowlan Freese added a comment -

          Is possible to create a public link to data in Google Drive that will work in IGB.
          For example: https://drive.google.com/uc?export=download&id=19AmDZxYkqVAgsK_o-fmlZry6zAQL5ExR&.bed

          There are a couple of caveats. The default shareable link provided by google is:
          https://drive.google.com/file/d/19AmDZxYkqVAgsK_o-fmlZry6zAQL5ExR/view?usp=sharing

          The id needs to be added to the default download url (https://drive.google.com/uc?export=download&id=UNIQUEFILEIDHERE). In order to get IGB to accept the file, it also needs the file extension. If &.FILEEXTENSION is added to the end of the URL, the download still works and IGB accepts the file.

          Show
          nfreese Nowlan Freese added a comment - Is possible to create a public link to data in Google Drive that will work in IGB. For example: https://drive.google.com/uc?export=download&id=19AmDZxYkqVAgsK_o-fmlZry6zAQL5ExR&.bed There are a couple of caveats. The default shareable link provided by google is: https://drive.google.com/file/d/19AmDZxYkqVAgsK_o-fmlZry6zAQL5ExR/view?usp=sharing The id needs to be added to the default download url ( https://drive.google.com/uc?export=download&id=UNIQUEFILEIDHERE ). In order to get IGB to accept the file, it also needs the file extension. If &.FILEEXTENSION is added to the end of the URL, the download still works and IGB accepts the file.
          Hide
          nfreese Nowlan Freese added a comment - - edited

          Was previously possible to store quickload in Dropbox . However, IGBF-1138 states that it may no longer be possible (this should be looked into, the issue is still open). UPDATE - It is no longer possible to host a Quickload in Dropbox.

          It is possible to create a public link to data in Dropbox that will work in IGB (including byte range requests).
          For example: https://dl.dropboxusercontent.com/s/7ida9n2p61lrtsa/Dm_genome_v1.0.bed

          Show
          nfreese Nowlan Freese added a comment - - edited Was previously possible to store quickload in Dropbox . However, IGBF-1138 states that it may no longer be possible (this should be looked into, the issue is still open). UPDATE - It is no longer possible to host a Quickload in Dropbox. It is possible to create a public link to data in Dropbox that will work in IGB (including byte range requests). For example: https://dl.dropboxusercontent.com/s/7ida9n2p61lrtsa/Dm_genome_v1.0.bed

            People

            • Assignee:
              stiwari8 Srishti Tiwari (Inactive)
              Reporter:
              nfreese Nowlan Freese
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: