Details

      Description

      Situation: In order to add CRAM file support (IGBF-1469) we need to update the HTSJDK dependency version. Currently we are using the artifact htsjdk-igb.

      Task: Investigate our use of the htsjdk-igb dependency. Determine how we can update this dependency.

      Github repo: https://github.com/lorainelab/htsjdk-igb

        Attachments

          Issue Links

            Activity

            Hide
            kgopu Kaushik Gopu added a comment - - edited

            For testing purpose, I replaced htsjdk-igb -2.16.2 with htsjdk-3.0.0 (official one) and observed the following issues:

            • The BigBed Data Handler implicitly downloading version 2.16.2 due to its dependency on IGV BigWig. Therefore, htsjdk version of IGV BigWig needs to be upgraded to 3.0.0
            • Updating BigWig/BigBed htsjdk-igb version would be the blocker for this one.
            Show
            kgopu Kaushik Gopu added a comment - - edited For testing purpose, I replaced htsjdk-igb -2.16.2 with htsjdk-3.0.0 (official one) and observed the following issues: The BigBed Data Handler implicitly downloading version 2.16.2 due to its dependency on IGV BigWig . Therefore, htsjdk version of IGV BigWig needs to be upgraded to 3.0.0 Updating BigWig/BigBed htsjdk-igb version would be the blocker for this one.
            Hide
            kgopu Kaushik Gopu added a comment - - edited

            Notes from discussion with Nowlan Freese:

            • So far, I have forked the htsjdk into my GitHub repository, implemented changes in the build.gradle file to use it as an IGB bundle, and deleted the SRA code. However, upon deleting SRA code, I had to comment out the code that uses SRA in some files. Finally, I am able to build the application without any compile-time or runtime errors.
            • Cherry-picking is not working really great. We have to deal with merge conflicts in many places, which is time consuming. For this reason, I have opted to implement changes manually.

            Questions to [~aloraine]

            • How would you like us to proceed with making htsjdk-3.0.0 available remotely? (We can discsuss this in a call with me and Nowlan Freese )
            • Do we really need to remove SRA code? Can we simply create a bundle without modifying/deleting code?
            Show
            kgopu Kaushik Gopu added a comment - - edited Notes from discussion with Nowlan Freese : So far, I have forked the htsjdk into my GitHub repository, implemented changes in the build.gradle file to use it as an IGB bundle, and deleted the SRA code. However, upon deleting SRA code, I had to comment out the code that uses SRA in some files. Finally, I am able to build the application without any compile-time or runtime errors. Cherry-picking is not working really great. We have to deal with merge conflicts in many places, which is time consuming. For this reason, I have opted to implement changes manually. Questions to [~aloraine] How would you like us to proceed with making htsjdk-3.0.0 available remotely? (We can discsuss this in a call with me and Nowlan Freese ) Do we really need to remove SRA code? Can we simply create a bundle without modifying/deleting code?
            Hide
            ann.loraine Ann Loraine added a comment -

            I don't believe you need to delete the SRA code.

            Can you test htsjdk as the authors have written it, without taking out any of their code?

            Show
            ann.loraine Ann Loraine added a comment - I don't believe you need to delete the SRA code. Can you test htsjdk as the authors have written it, without taking out any of their code?
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            We noticed during a meeting after scrum the following issues:

            • The main project is developing on one branch only - master (see: https://github.com/samtools/htsjdk/tree/master)
            • They have tags indicating specific releases. For example, there are tags such as 3.0.0, 3.0.1, 3.0.2, ... , 3.0.5
            • We are going to use the code/commit tagged as 3.0.5 (the last 3.0.x commit) for our project
            • This appears to be the last commit of the htsjdk project that supports JDK8
            • Unfortunately there is no way we can think to make a change to 3.0.5 on the upstream project because "3.0.5" is nothing but a commit on a chain of commits belonging to branch "master"
            • We looked into it because we thought we might be able to add the bundle-making logic to the upstream project's build and thus never have to release the bundle ourselves because the upstream project's released artifacts would therefore always be bundles
            Show
            ann.loraine Ann Loraine added a comment - - edited We noticed during a meeting after scrum the following issues: The main project is developing on one branch only - master (see: https://github.com/samtools/htsjdk/tree/master ) They have tags indicating specific releases. For example, there are tags such as 3.0.0, 3.0.1, 3.0.2, ... , 3.0.5 We are going to use the code/commit tagged as 3.0.5 (the last 3.0.x commit) for our project This appears to be the last commit of the htsjdk project that supports JDK8 Unfortunately there is no way we can think to make a change to 3.0.5 on the upstream project because "3.0.5" is nothing but a commit on a chain of commits belonging to branch "master" We looked into it because we thought we might be able to add the bundle-making logic to the upstream project's build and thus never have to release the bundle ourselves because the upstream project's released artifacts would therefore always be bundles
            Hide
            nfreese Nowlan Freese added a comment -

            We were running into an issue with BaiFileSymLoader.java and BaiSymLoader.java not working with the newer version of the htsjdk-igb. It turns out that the code from the lorainelab htsjdk-igb github page is not up to date. It appears that Charan's github page has the code: https://github.com/VallapuCharan/htsjdk-igb/commits/igb-2.16.3

            Next step is to implement Charan's code changes to the htsjdk-igb 3.0.0 version.

            Show
            nfreese Nowlan Freese added a comment - We were running into an issue with BaiFileSymLoader.java and BaiSymLoader.java not working with the newer version of the htsjdk-igb. It turns out that the code from the lorainelab htsjdk-igb github page is not up to date. It appears that Charan's github page has the code: https://github.com/VallapuCharan/htsjdk-igb/commits/igb-2.16.3 Next step is to implement Charan's code changes to the htsjdk-igb 3.0.0 version.
            Hide
            kgopu Kaushik Gopu added a comment - - edited
            • Implemented charan's code changes to the htsjdk-igb 3.0.0 version. No issues with BaiFileSymLoader.java and BaiSymLoader.java now.
            • The application can now be built and run without encountering any compile-time or runtime errors.
            • However, when a BAI file is provided to the application, and there is an equivalent BAM file in the same location as the BAI file, the application throws an exception. This behavior is consistent in the stable release as well. While not a critical issue, it would be beneficial to handle this scenario by displaying a user-friendly pop-up message.
            Show
            kgopu Kaushik Gopu added a comment - - edited Implemented charan's code changes to the htsjdk-igb 3.0.0 version. No issues with BaiFileSymLoader.java and BaiSymLoader.java now. The application can now be built and run without encountering any compile-time or runtime errors. However, when a BAI file is provided to the application, and there is an equivalent BAM file in the same location as the BAI file, the application throws an exception. This behavior is consistent in the stable release as well. While not a critical issue, it would be beneficial to handle this scenario by displaying a user-friendly pop-up message.
            Hide
            kgopu Kaushik Gopu added a comment - - edited

            Testing Instructions:

            • Clone this repository into your local and checkout to branch igb_main-3.0.0.
            • Run:
              command

              gradle clean build -x test

            • A folder named build will appear after running the above command. Look for htsjdk-igb-3.0.0.jar in build/libs folder.
            • Install the artifact in your local maven repository using the below command:
              Installing artifact locally

              mvn install:install-file -Dfile="<path to jar file>" -DgroupId="htsjdk-igb" -DartifactId="htsjdk-igb" -Dversion="3.0.0" -Dpackaging="jar" -DgeneratePom="true"

            • Make sure you delete the old version of htsjdk before installing the latest one

              Install BigWig Artifact: BigWig uses the htsjdk library. Since IGB depends on BigWig, the htsjdk version in BigWig needs to be upgraded to 3.0.0 to avoid dependency conflicts.

            • Clone BigWig into your local.
            • In build.gradle file, change htsjdk-igb version to 3.0.0
              • May also need to add mavenLocal() under repositories in the build.gradle file
            • Run: gradle build -x test
            • Install the artifact to your local maven repository using the below command (make sure you delete the existing bigwig artifact before installing the below one):
              Installing BigWig artifact

              mvn install:install-file -Dfile="<path>/bigwig-2.0.0.jar" -DgroupId="org.broad.igv" -DartifactId="bigwig" -Dversion="2.0.0" -Dpackaging="jar" -DgeneratePom="true"


            Testing IGB:

            Show
            kgopu Kaushik Gopu added a comment - - edited Testing Instructions: Clone this repository into your local and checkout to branch igb_main-3.0.0 . Run: command gradle clean build -x test A folder named build will appear after running the above command. Look for htsjdk-igb-3.0.0.jar in build/libs folder. Install the artifact in your local maven repository using the below command: Installing artifact locally mvn install:install-file -Dfile="<path to jar file>" -DgroupId="htsjdk-igb" -DartifactId="htsjdk-igb" -Dversion="3.0.0" -Dpackaging="jar" -DgeneratePom="true" Make sure you delete the old version of htsjdk before installing the latest one Install BigWig Artifact: BigWig uses the htsjdk library. Since IGB depends on BigWig, the htsjdk version in BigWig needs to be upgraded to 3.0.0 to avoid dependency conflicts. Clone BigWig into your local. In build.gradle file, change htsjdk-igb version to 3.0.0 May also need to add mavenLocal() under repositories in the build.gradle file Run: gradle build -x test Install the artifact to your local maven repository using the below command (make sure you delete the existing bigwig artifact before installing the below one): Installing BigWig artifact mvn install:install-file -Dfile="<path>/bigwig-2.0.0.jar" -DgroupId="org.broad.igv" -DartifactId="bigwig" -Dversion="2.0.0" -Dpackaging="jar" -DgeneratePom="true" Testing IGB: Switch to branch: https://bitbucket.org/kaushik-gopu/kgopu_integrated-genome-browser/branch/IGBF-3464 Run mvn clean install Test IGB by inputting different track formats such as BAI, BED, BAM and also keep monitoring logs while testing.
            Hide
            nfreese Nowlan Freese added a comment - - edited

            Tested following above instructions on Mac:

            • Able to add and load data from a bam file from IGB Quickload.
            • Able to add and load data (visualize) a bai file.
            • Able to create a depth graph from a bam file.
            • Able to add and load data from a local bam file.
            • Able to add and load data from a local bed file.
            • Able to add and load data from a local sam file.

            Kaushik Gopu - everything seems to be working as expected that I can think of. Please move forward with pull requests once artifacts have been deployed.

            Show
            nfreese Nowlan Freese added a comment - - edited Tested following above instructions on Mac: Able to add and load data from a bam file from IGB Quickload. Able to add and load data (visualize) a bai file. Able to create a depth graph from a bam file. Able to add and load data from a local bam file. Able to add and load data from a local bed file. Able to add and load data from a local sam file. Kaushik Gopu - everything seems to be working as expected that I can think of. Please move forward with pull requests once artifacts have been deployed.
            Hide
            ann.loraine Ann Loraine added a comment -

            The htsjdk artifact can't be deployed just yet because the bundle symbolic name is wrong. It's showing up as "htsjdk" in the MANIFEST but it should instead be "htsjdk-igb"

            Moving back to "To-Do"

            Show
            ann.loraine Ann Loraine added a comment - The htsjdk artifact can't be deployed just yet because the bundle symbolic name is wrong. It's showing up as "htsjdk" in the MANIFEST but it should instead be "htsjdk-igb" Moving back to "To-Do"
            Hide
            ann.loraine Ann Loraine added a comment -

            Bundle, javadoc and source file all deployed and ready for testing.

            See: https://nexus.bioviz.org/#browse/browse:maven-releases:htsjdk-igb

            Show
            ann.loraine Ann Loraine added a comment - Bundle, javadoc and source file all deployed and ready for testing. See: https://nexus.bioviz.org/#browse/browse:maven-releases:htsjdk-igb
            Hide
            nfreese Nowlan Freese added a comment -

            Bundle appears to be deployed successfully. The bundle is successfully retrieved and installed in my local .m2 when building IGB and specifying htsjdk-igb version 3.0.5.

            Closing ticket.

            Show
            nfreese Nowlan Freese added a comment - Bundle appears to be deployed successfully. The bundle is successfully retrieved and installed in my local .m2 when building IGB and specifying htsjdk-igb version 3.0.5. Closing ticket.
            Hide
            kgopu Kaushik Gopu added a comment - - edited

            Updated IGB's htsjdk and bigWig versions to 3.0.5 and 2.0.1, respectively in the IGB project pom.
            Changes are available to in this commit
            Moving to first level review

            Show
            kgopu Kaushik Gopu added a comment - - edited Updated IGB's htsjdk and bigWig versions to 3.0.5 and 2.0.1, respectively in the IGB project pom. Changes are available to in this commit Moving to first level review
            Hide
            nfreese Nowlan Freese added a comment -

            Commit looks good, ready for pull request.

            Note that it would be best to deploy the changes to Bigwig (see IGBF-3465) prior to merging this pull request.

            Show
            nfreese Nowlan Freese added a comment - Commit looks good, ready for pull request. Note that it would be best to deploy the changes to Bigwig (see IGBF-3465 ) prior to merging this pull request.
            Hide
            kgopu Kaushik Gopu added a comment -
            • PR has been submitted.
            • It would be best to deploy bigWig (IGBF-3464) prior to merging this PR
            Show
            kgopu Kaushik Gopu added a comment - PR has been submitted. It would be best to deploy bigWig ( IGBF-3464 ) prior to merging this PR
            Hide
            ann.loraine Ann Loraine added a comment -

            PR is merged and IGB is built, with no errors.

            attn: Kaushik Gopu

            Show
            ann.loraine Ann Loraine added a comment - PR is merged and IGB is built, with no errors. attn: Kaushik Gopu
            Hide
            nfreese Nowlan Freese added a comment -

            Tested on Mac using strategy described in one of my previous comments. Everything is working correctly, closing ticket.

            Show
            nfreese Nowlan Freese added a comment - Tested on Mac using strategy described in one of my previous comments. Everything is working correctly, closing ticket.

              People

              • Assignee:
                kgopu Kaushik Gopu
                Reporter:
                nfreese Nowlan Freese
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: