Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-32

Don't replace spaces with underscore characters when viewing files from Galaxy.

    Details

    • Type: Improvement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
    • Story Points:
      4
    • Sprint:
      Summer 2019 Sprint 11, Summer 2019 Sprint 12, Fall 2019 Sprint 1, Fall 2019 Sprint 2, Fall 2019 Sprint 3, Fall 4 : 30 Sep to 11 Oct, Fall 5 : 14 Oct to 25 Oct

      Description

      Kick this up to 8.2 if not feasible to do safely for 8.1 release.

      It would be nice if the name of tracks imported from Galaxy would EXACTLY match the name of the data set as shown in the Galaxy interface. Currently, names that contain spaces are being replaced by underscore characters. It would be better if the names were identical.

        Attachments

          Issue Links

            Activity

            Hide
            ann.loraine Ann Loraine added a comment - - edited

            This will require modifying the viewers XML in Galaxy. Let's do this after the GCC 2019 meeting.

            Show
            ann.loraine Ann Loraine added a comment - - edited This will require modifying the viewers XML in Galaxy. Let's do this after the GCC 2019 meeting.
            Hide
            noor91zahara Noor Zahara (Inactive) added a comment -

            [~aloraine] - Since this is an open source project, should I directly push my changes ? Could you suggest on this ?

            Show
            noor91zahara Noor Zahara (Inactive) added a comment - [~aloraine] - Since this is an open source project, should I directly push my changes ? Could you suggest on this ?
            Hide
            ann.loraine Ann Loraine added a comment -

            To make changes to Galaxy code base:

            • Read contribution and coding standard guidelines for the Galaxy project.
            • Look over recent pull requests to get a sense of how those are handled.
            • Review previous Loraine Lab contributions to Galaxy code base (there are a few - probably from 2014?)
            • Follow their current Galaxy project instructions for contributions, coding standards.
            • If you have questions, ask for guidance via their recommended channel. (Probably will be a gitter channel - like Slack.)

            If you have changes to the Galaxy code:

            • Set up and provision a test Galaxy site our testers can use to validate the changes. (Use EC2 or other semi-permanent host)
            • Provide detail instructions for testers on how to validate/test the changes.
            Show
            ann.loraine Ann Loraine added a comment - To make changes to Galaxy code base: Read contribution and coding standard guidelines for the Galaxy project. Look over recent pull requests to get a sense of how those are handled. Review previous Loraine Lab contributions to Galaxy code base (there are a few - probably from 2014?) Follow their current Galaxy project instructions for contributions, coding standards. If you have questions, ask for guidance via their recommended channel. (Probably will be a gitter channel - like Slack.) If you have changes to the Galaxy code: Set up and provision a test Galaxy site our testers can use to validate the changes. (Use EC2 or other semi-permanent host) Provide detail instructions for testers on how to validate/test the changes.
            Hide
            noor91zahara Noor Zahara (Inactive) added a comment -

            I found out that BBFileReader class imported from package org.broad.igv.bbfile is throwing an exception while uploading bigwig and bigbed files.

            Error Message : Error reading R+ tree header: bad magic = -2003829722

            I found out the java file which is throwing an exception and it looks like it is due to some RPTree header not getting set properly.

            https://www.javatips.net/api/igv-master/src/org/broad/igv/bbfile/RPTree.java

            [~aloraine] - I was wondering if you ever have come across the above error message while using galaxy ?

            Show
            noor91zahara Noor Zahara (Inactive) added a comment - I found out that BBFileReader class imported from package org.broad.igv.bbfile is throwing an exception while uploading bigwig and bigbed files. Error Message : Error reading R+ tree header: bad magic = -2003829722 I found out the java file which is throwing an exception and it looks like it is due to some RPTree header not getting set properly. https://www.javatips.net/api/igv-master/src/org/broad/igv/bbfile/RPTree.java [~aloraine] - I was wondering if you ever have come across the above error message while using galaxy ?
            Hide
            noor91zahara Noor Zahara (Inactive) added a comment -

            I have also raised a ticket with galaxy for getting some suggestions

            https://github.com/galaxyproject/galaxy/issues/8401

            Show
            noor91zahara Noor Zahara (Inactive) added a comment - I have also raised a ticket with galaxy for getting some suggestions https://github.com/galaxyproject/galaxy/issues/8401
            Hide
            ann.loraine Ann Loraine added a comment -

            Please perform a regression test to determine if the same error occurs with IGB 9.0.2.

            Show
            ann.loraine Ann Loraine added a comment - Please perform a regression test to determine if the same error occurs with IGB 9.0.2.
            Hide
            noor91zahara Noor Zahara (Inactive) added a comment -

            I downloaded IGB 9.0.2 and facing the same issue.

            Show
            noor91zahara Noor Zahara (Inactive) added a comment - I downloaded IGB 9.0.2 and facing the same issue.
            Hide
            ann.loraine Ann Loraine added a comment -

            Please give step-by-step description of how to reproduce the bug. (Not sure what you mean by "upload" because IGB is not uploading anything.)

            Show
            ann.loraine Ann Loraine added a comment - Please give step-by-step description of how to reproduce the bug. (Not sure what you mean by "upload" because IGB is not uploading anything.)
            Hide
            noor91zahara Noor Zahara (Inactive) added a comment -

            Steps to reproduce the issue -

            1. Upload bigwig/bigbed/bed file to galaxy
            2. Click on Display the files in IGB

            The error can be viewed on IGB Console.

            Show
            noor91zahara Noor Zahara (Inactive) added a comment - Steps to reproduce the issue - 1. Upload bigwig/bigbed/bed file to galaxy 2. Click on Display the files in IGB The error can be viewed on IGB Console.
            Hide
            ann.loraine Ann Loraine added a comment -

            Please clarify:

            • Which Galaxy?
            • Which bigwig?
            Show
            ann.loraine Ann Loraine added a comment - Please clarify: Which Galaxy? Which bigwig?
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Please perform release testing protocol here:

            If you perform these actions using 9.0.1 or 9.1.0 (latest master) do you observe an error?

            Show
            ann.loraine Ann Loraine added a comment - - edited Please perform release testing protocol here: https://wiki.transvar.org/display/ITD/Galaxy If you perform these actions using 9.0.1 or 9.1.0 (latest master) do you observe an error?
            Hide
            sameer Sameer Shanbhag (Inactive) added a comment -

            Can be tested on the following Galaxy Instance:

            https://18.204.227.154/

            Show
            sameer Sameer Shanbhag (Inactive) added a comment - Can be tested on the following Galaxy Instance: https://18.204.227.154/
            Hide
            noor91zahara Noor Zahara (Inactive) added a comment - - edited

            Code changes for galaxy : https://github.com/galaxyproject/galaxy/compare/dev...nzahara1:IGBF-32-C
            Code changes in IGB : https://bitbucket.org/noorzahara/integrated-genome-browser-local1/branch/IGBF-32#diff

            Login credentials for galaxy instance:
            noor91zahara@gmail.com
            password

            Testing procedure

            1. Files having spaces have been uploaded.
            2. View the same on IGB.
            3. File names should be the same as the ones being uploaded on galaxy.

            Show
            noor91zahara Noor Zahara (Inactive) added a comment - - edited Code changes for galaxy : https://github.com/galaxyproject/galaxy/compare/dev...nzahara1:IGBF-32-C Code changes in IGB : https://bitbucket.org/noorzahara/integrated-genome-browser-local1/branch/IGBF-32#diff Login credentials for galaxy instance: noor91zahara@gmail.com password Testing procedure 1. Files having spaces have been uploaded. 2. View the same on IGB. 3. File names should be the same as the ones being uploaded on galaxy.
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            I have reviewed the code and it looks file. User is allowed to upload files which has spaces in their name instead of replacing the white space in the file names with underscores.
            I have checked the same thing on galaxy but I am not sure how to check it on the IGB platform.
            Noor Zahara, please let me know.
            Thanks.

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - I have reviewed the code and it looks file. User is allowed to upload files which has spaces in their name instead of replacing the white space in the file names with underscores. I have checked the same thing on galaxy but I am not sure how to check it on the IGB platform. Noor Zahara , please let me know. Thanks.
            Hide
            noor91zahara Noor Zahara (Inactive) added a comment -

            Prutha Kulkarni - Need to open galaxy instance - https://18.204.227.154/

            Click on display in IGB under each file.

            Show
            noor91zahara Noor Zahara (Inactive) added a comment - Prutha Kulkarni - Need to open galaxy instance - https://18.204.227.154/ Click on display in IGB under each file.
            Hide
            ann.loraine Ann Loraine added a comment -

            You need to explain how to access the files.

            Show
            ann.loraine Ann Loraine added a comment - You need to explain how to access the files.
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            Noor Zahara
            It is working as expected. But, currently its working with the domain name and not with the IP. I am waiting for Sameer Shanbhag to setup the galaxy instance with new host. Once that is done, will test it again and move it to Ready for Pull Request.

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - Noor Zahara It is working as expected. But, currently its working with the domain name and not with the IP. I am waiting for Sameer Shanbhag to setup the galaxy instance with new host. Once that is done, will test it again and move it to Ready for Pull Request.
            Hide
            prutha Prutha Kulkarni (Inactive) added a comment -

            I have tested it on usegalaxy.bioviz.org and its working fine now. Moving the ticket to ready for pull request.

            Show
            prutha Prutha Kulkarni (Inactive) added a comment - I have tested it on usegalaxy.bioviz.org and its working fine now. Moving the ticket to ready for pull request.
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Nowlan Freese - could you also check the functionality?

            We need to be extra careful that the functionality is working as expected. Since you are super familiar with Galaxy, it would be very good if you could also take a look.

            Basically, we need to double-check that this is working properly with each of the different file types IGB can open from Galaxy.

            Show
            ann.loraine Ann Loraine added a comment - - edited Nowlan Freese - could you also check the functionality? We need to be extra careful that the functionality is working as expected. Since you are super familiar with Galaxy, it would be very good if you could also take a look. Basically, we need to double-check that this is working properly with each of the different file types IGB can open from Galaxy.
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Also, please check behavior with released IGB 9.0.2.

            I don't think it will work with the changes made to the Galaxy code base, but the modifications made by Noor Zahara to IGB itself (see topic branch) should be backward compatible with the current Galaxy code deployed on usegalaxy.org. Basically, it looks like the new IGB code is simply doing a better job of handling special characters in Galaxy data URLs.

            Nowlan Freese can you confirm?

            Show
            ann.loraine Ann Loraine added a comment - - edited Also, please check behavior with released IGB 9.0.2. I don't think it will work with the changes made to the Galaxy code base, but the modifications made by Noor Zahara to IGB itself (see topic branch) should be backward compatible with the current Galaxy code deployed on usegalaxy.org. Basically, it looks like the new IGB code is simply doing a better job of handling special characters in Galaxy data URLs. Nowlan Freese can you confirm?
            Hide
            nfreese Nowlan Freese added a comment - - edited

            To test, I uploaded the human data from the IGB smoke test quickload to both usegalaxy.bioviz.org and usegalaxy.org. Data were tested against Noor's IGBF-32 branch and against IGB 9.0.2.

            Regarding the cross compatibility of IGB and Galaxy versions:
            The current version of usegalaxy.org will work with IGBF-32. As expected, spaces are still converted to underscores (old galaxy works with new IGB). However, the new version of Galaxy (usegalaxy.bioviz.org) will not work with IGB 9.0.2 (new galaxy does not work with old IGB). Suggest releasing IGB changes first, and giving users some amount of time to upgrade before pushing the new changes to Galaxy.

            Wig, GTF, BED, and BAM behaved as expected and were properly formatted within IGB with spaces that matched the names from usegalaxy.bioviz.org. Biwig, Bigbed, and Bedgraph behaved as expected and initially showed spaces within the names correctly, but upon loading data the names changed. Most likely this behavior has been fixed with IGBF-1933, but IGBF-32 does not have these changes. These should be retested once the IGBF-32 changes have been merged to master.

            Sidenote:
            Galaxy did not enable the View in IGB link for VCF, GFF, narrowpeak, broadpeak, or SAM files. VCF and GFF were available to view in IGV. Narrowpeak, broadpeak and SAM files had no links to view in any genome browser.

            Show
            nfreese Nowlan Freese added a comment - - edited To test, I uploaded the human data from the IGB smoke test quickload to both usegalaxy.bioviz.org and usegalaxy.org. Data were tested against Noor's IGBF-32 branch and against IGB 9.0.2. Regarding the cross compatibility of IGB and Galaxy versions: The current version of usegalaxy.org will work with IGBF-32 . As expected, spaces are still converted to underscores (old galaxy works with new IGB). However, the new version of Galaxy (usegalaxy.bioviz.org) will not work with IGB 9.0.2 (new galaxy does not work with old IGB). Suggest releasing IGB changes first, and giving users some amount of time to upgrade before pushing the new changes to Galaxy. Wig, GTF, BED, and BAM behaved as expected and were properly formatted within IGB with spaces that matched the names from usegalaxy.bioviz.org. Biwig, Bigbed, and Bedgraph behaved as expected and initially showed spaces within the names correctly, but upon loading data the names changed. Most likely this behavior has been fixed with IGBF-1933 , but IGBF-32 does not have these changes. These should be retested once the IGBF-32 changes have been merged to master. Sidenote: Galaxy did not enable the View in IGB link for VCF, GFF, narrowpeak, broadpeak, or SAM files. VCF and GFF were available to view in IGV. Narrowpeak, broadpeak and SAM files had no links to view in any genome browser.
            Hide
            ann.loraine Ann Loraine added a comment -

            Noor Zahara: Please do this:

            1) Please submit PR for IGBF-32 to lorainelab master branch.

            2) Do not submit PR for branch https://github.com/galaxyproject/galaxy/compare/dev...nzahara1:IGBF-32-C. I will make a separate ticket regarding next steps for your Galaxy changes.

            Show
            ann.loraine Ann Loraine added a comment - Noor Zahara : Please do this: 1) Please submit PR for IGBF-32 to lorainelab master branch. 2) Do not submit PR for branch https://github.com/galaxyproject/galaxy/compare/dev...nzahara1:IGBF-32-C . I will make a separate ticket regarding next steps for your Galaxy changes.
            Show
            noor91zahara Noor Zahara (Inactive) added a comment - I have created a pull request for IGBF-32 https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/731/igbf-32/diff
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Pull request is merged to master branch. New installers have built and are deployed to BioViz.org as "early access" IGB 9.1.0.

            Nowlan Freese when you have some time, do please repeat testing you did using topic branch installer previously but now using this newly built 9.1.0 installer.

            Show
            ann.loraine Ann Loraine added a comment - - edited Pull request is merged to master branch. New installers have built and are deployed to BioViz.org as "early access" IGB 9.1.0. Nowlan Freese when you have some time, do please repeat testing you did using topic branch installer previously but now using this newly built 9.1.0 installer.
            Hide
            nfreese Nowlan Freese added a comment -

            The usegalaxy.bioviz.org server seems to be down.

            Show
            nfreese Nowlan Freese added a comment - The usegalaxy.bioviz.org server seems to be down.
            Hide
            ann.loraine Ann Loraine added a comment -

            I have restarted it. It is now available.
            Nowlan Freese

            Show
            ann.loraine Ann Loraine added a comment - I have restarted it. It is now available. Nowlan Freese
            Hide
            nfreese Nowlan Freese added a comment - - edited

            Testing on early access IGB as of September 25, 2019 and usegalaxy.bioviz.org.

            All file formats except for bigbed are shown correctly with spaces.

            Bigbed is displaying spaces correctly before the data are loaded. Once Load Data is clicked spaces are replaced with "%20".

            Show
            nfreese Nowlan Freese added a comment - - edited Testing on early access IGB as of September 25, 2019 and usegalaxy.bioviz.org. All file formats except for bigbed are shown correctly with spaces. Bigbed is displaying spaces correctly before the data are loaded. Once Load Data is clicked spaces are replaced with "%20".
            Hide
            noor91zahara Noor Zahara (Inactive) added a comment - - edited

            The above mentioned bug has been fixed.
            Code changes can be viewed - https://bitbucket.org/noorzahara/integrated-genome-browser-local1/branch/IGBF-32#diff

            Note from Dr. Loraine (added Nov 2019): Branch is merged into IGB 9.1

            Show
            noor91zahara Noor Zahara (Inactive) added a comment - - edited The above mentioned bug has been fixed. Code changes can be viewed - https://bitbucket.org/noorzahara/integrated-genome-browser-local1/branch/IGBF-32#diff Note from Dr. Loraine (added Nov 2019): Branch is merged into IGB 9.1
            Hide
            nfreese Nowlan Freese added a comment - - edited

            The spaces are now showing correctly in bigbed, but when I click Load Data the file extension is removed from the name.

            Edited:
            After talking with Noor, the bigbed file extension issue does not seem to be related to her changes. The release version of IGB also has the behavior of removing bigbed file extensions. I'm not sure why this is the case, as this is not consistent to how other file types are treated.

            Show
            nfreese Nowlan Freese added a comment - - edited The spaces are now showing correctly in bigbed, but when I click Load Data the file extension is removed from the name. Edited: After talking with Noor, the bigbed file extension issue does not seem to be related to her changes. The release version of IGB also has the behavior of removing bigbed file extensions. I'm not sure why this is the case, as this is not consistent to how other file types are treated.
            Hide
            noor91zahara Noor Zahara (Inactive) added a comment - - edited

            I investigated on how to support backward compatibility to older version of IGB.

            I tried to replace spaces/ non-alphabetic characters in the file name with "" in bioviz before sending it to the older version of IGB ( replicating the functionality of galaxy before my changes). But the issue here is that IGB tries to validate the url (modified url via bioviz) and figures out that it is invalid since that file name does not exist in galaxy. (In galaxy the filename is stored without "" .)

            Show
            noor91zahara Noor Zahara (Inactive) added a comment - - edited I investigated on how to support backward compatibility to older version of IGB. I tried to replace spaces/ non-alphabetic characters in the file name with " " in bioviz before sending it to the older version of IGB ( replicating the functionality of galaxy before my changes). But the issue here is that IGB tries to validate the url (modified url via bioviz) and figures out that it is invalid since that file name does not exist in galaxy. (In galaxy the filename is stored without " " .)
            Hide
            noor91zahara Noor Zahara (Inactive) added a comment -

            Closing this ticket and creating a new one to investigate about the IGB backward compatibility with these changes of galaxy.

            Show
            noor91zahara Noor Zahara (Inactive) added a comment - Closing this ticket and creating a new one to investigate about the IGB backward compatibility with these changes of galaxy.

              People

              • Assignee:
                noor91zahara Noor Zahara (Inactive)
                Reporter:
                ann.loraine Ann Loraine
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: