Details

      Description

      Situation: In order to add CRAM file support (IGBF-1469) we need to update the HTSJDK dependency version. Currently we are using the artifact htsjdk-igb.

      Task: Investigate our use of the htsjdk-igb dependency. Determine how we can update this dependency.

      Github repo: https://github.com/lorainelab/htsjdk-igb

        Attachments

          Issue Links

            Activity

            nfreese Nowlan Freese created issue -
            nfreese Nowlan Freese made changes -
            Field Original Value New Value
            Epic Link IGBF-1531 [ 17617 ]
            nfreese Nowlan Freese made changes -
            Link This issue blocks IGBF-1469 [ IGBF-1469 ]
            nfreese Nowlan Freese made changes -
            Link This issue relates to IGBF-1418 [ IGBF-1418 ]
            nfreese Nowlan Freese made changes -
            Link This issue is blocked by IGBF-3465 [ IGBF-3465 ]
            Hide
            kgopu Kaushik Gopu added a comment - - edited

            For testing purpose, I replaced htsjdk-igb -2.16.2 with htsjdk-3.0.0 (official one) and observed the following issues:

            • The BigBed Data Handler implicitly downloading version 2.16.2 due to its dependency on IGV BigWig. Therefore, htsjdk version of IGV BigWig needs to be upgraded to 3.0.0
            • Updating BigWig/BigBed htsjdk-igb version would be the blocker for this one.
            Show
            kgopu Kaushik Gopu added a comment - - edited For testing purpose, I replaced htsjdk-igb -2.16.2 with htsjdk-3.0.0 (official one) and observed the following issues: The BigBed Data Handler implicitly downloading version 2.16.2 due to its dependency on IGV BigWig . Therefore, htsjdk version of IGV BigWig needs to be upgraded to 3.0.0 Updating BigWig/BigBed htsjdk-igb version would be the blocker for this one.
            nfreese Nowlan Freese made changes -
            Sprint Fall 3 2023 Oct 2 [ 179 ] Fall 4 2023 Oct 16 [ 180 ]
            nfreese Nowlan Freese made changes -
            Epic Link IGBF-1531 [ 17617 ] IGBF-3467 [ 22628 ]
            nfreese Nowlan Freese made changes -
            Description Situation: In order to add CRAM file support (IGBF-1469) we need to update the HTSJDK dependency version. Currently we are using the artifact htsjdk-igb.

            Task: Investigate our use of the htsjd-igb dependency. Determine how we can update this dependency.
            Situation: In order to add CRAM file support (IGBF-1469) we need to update the HTSJDK dependency version. Currently we are using the artifact htsjdk-igb.

            Task: Investigate our use of the htsjd-igb dependency. Determine how we can update this dependency.

            Github repo: https://github.com/lorainelab/htsjdk-igb
            nfreese Nowlan Freese made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            nfreese Nowlan Freese made changes -
            Link This issue blocks IGBF-3478 [ IGBF-3478 ]
            Hide
            kgopu Kaushik Gopu added a comment - - edited

            Notes from discussion with Nowlan Freese:

            • So far, I have forked the htsjdk into my GitHub repository, implemented changes in the build.gradle file to use it as an IGB bundle, and deleted the SRA code. However, upon deleting SRA code, I had to comment out the code that uses SRA in some files. Finally, I am able to build the application without any compile-time or runtime errors.
            • Cherry-picking is not working really great. We have to deal with merge conflicts in many places, which is time consuming. For this reason, I have opted to implement changes manually.

            Questions to [~aloraine]

            • How would you like us to proceed with making htsjdk-3.0.0 available remotely? (We can discsuss this in a call with me and Nowlan Freese )
            • Do we really need to remove SRA code? Can we simply create a bundle without modifying/deleting code?
            Show
            kgopu Kaushik Gopu added a comment - - edited Notes from discussion with Nowlan Freese : So far, I have forked the htsjdk into my GitHub repository, implemented changes in the build.gradle file to use it as an IGB bundle, and deleted the SRA code. However, upon deleting SRA code, I had to comment out the code that uses SRA in some files. Finally, I am able to build the application without any compile-time or runtime errors. Cherry-picking is not working really great. We have to deal with merge conflicts in many places, which is time consuming. For this reason, I have opted to implement changes manually. Questions to [~aloraine] How would you like us to proceed with making htsjdk-3.0.0 available remotely? (We can discsuss this in a call with me and Nowlan Freese ) Do we really need to remove SRA code? Can we simply create a bundle without modifying/deleting code?
            Hide
            ann.loraine Ann Loraine added a comment -

            I don't believe you need to delete the SRA code.

            Can you test htsjdk as the authors have written it, without taking out any of their code?

            Show
            ann.loraine Ann Loraine added a comment - I don't believe you need to delete the SRA code. Can you test htsjdk as the authors have written it, without taking out any of their code?
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            We noticed during a meeting after scrum the following issues:

            • The main project is developing on one branch only - master (see: https://github.com/samtools/htsjdk/tree/master)
            • They have tags indicating specific releases. For example, there are tags such as 3.0.0, 3.0.1, 3.0.2, ... , 3.0.5
            • We are going to use the code/commit tagged as 3.0.5 (the last 3.0.x commit) for our project
            • This appears to be the last commit of the htsjdk project that supports JDK8
            • Unfortunately there is no way we can think to make a change to 3.0.5 on the upstream project because "3.0.5" is nothing but a commit on a chain of commits belonging to branch "master"
            • We looked into it because we thought we might be able to add the bundle-making logic to the upstream project's build and thus never have to release the bundle ourselves because the upstream project's released artifacts would therefore always be bundles
            Show
            ann.loraine Ann Loraine added a comment - - edited We noticed during a meeting after scrum the following issues: The main project is developing on one branch only - master (see: https://github.com/samtools/htsjdk/tree/master ) They have tags indicating specific releases. For example, there are tags such as 3.0.0, 3.0.1, 3.0.2, ... , 3.0.5 We are going to use the code/commit tagged as 3.0.5 (the last 3.0.x commit) for our project This appears to be the last commit of the htsjdk project that supports JDK8 Unfortunately there is no way we can think to make a change to 3.0.5 on the upstream project because "3.0.5" is nothing but a commit on a chain of commits belonging to branch "master" We looked into it because we thought we might be able to add the bundle-making logic to the upstream project's build and thus never have to release the bundle ourselves because the upstream project's released artifacts would therefore always be bundles
            Hide
            nfreese Nowlan Freese added a comment -

            We were running into an issue with BaiFileSymLoader.java and BaiSymLoader.java not working with the newer version of the htsjdk-igb. It turns out that the code from the lorainelab htsjdk-igb github page is not up to date. It appears that Charan's github page has the code: https://github.com/VallapuCharan/htsjdk-igb/commits/igb-2.16.3

            Next step is to implement Charan's code changes to the htsjdk-igb 3.0.0 version.

            Show
            nfreese Nowlan Freese added a comment - We were running into an issue with BaiFileSymLoader.java and BaiSymLoader.java not working with the newer version of the htsjdk-igb. It turns out that the code from the lorainelab htsjdk-igb github page is not up to date. It appears that Charan's github page has the code: https://github.com/VallapuCharan/htsjdk-igb/commits/igb-2.16.3 Next step is to implement Charan's code changes to the htsjdk-igb 3.0.0 version.
            Hide
            kgopu Kaushik Gopu added a comment - - edited
            • Implemented charan's code changes to the htsjdk-igb 3.0.0 version. No issues with BaiFileSymLoader.java and BaiSymLoader.java now.
            • The application can now be built and run without encountering any compile-time or runtime errors.
            • However, when a BAI file is provided to the application, and there is an equivalent BAM file in the same location as the BAI file, the application throws an exception. This behavior is consistent in the stable release as well. While not a critical issue, it would be beneficial to handle this scenario by displaying a user-friendly pop-up message.
            Show
            kgopu Kaushik Gopu added a comment - - edited Implemented charan's code changes to the htsjdk-igb 3.0.0 version. No issues with BaiFileSymLoader.java and BaiSymLoader.java now. The application can now be built and run without encountering any compile-time or runtime errors. However, when a BAI file is provided to the application, and there is an equivalent BAM file in the same location as the BAI file, the application throws an exception. This behavior is consistent in the stable release as well. While not a critical issue, it would be beneficial to handle this scenario by displaying a user-friendly pop-up message.
            ann.loraine Ann Loraine made changes -
            Sprint Fall 4 2023 Oct 16 [ 180 ] Fall 4 2023 Oct 16, Fall 5 [ 180, 181 ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            Hide
            kgopu Kaushik Gopu added a comment - - edited

            Testing Instructions:

            • Clone this repository into your local and checkout to branch igb_main-3.0.0.
            • Run:
              command

              gradle clean build -x test

            • A folder named build will appear after running the above command. Look for htsjdk-igb-3.0.0.jar in build/libs folder.
            • Install the artifact in your local maven repository using the below command:
              Installing artifact locally

              mvn install:install-file -Dfile="<path to jar file>" -DgroupId="htsjdk-igb" -DartifactId="htsjdk-igb" -Dversion="3.0.0" -Dpackaging="jar" -DgeneratePom="true"

            • Make sure you delete the old version of htsjdk before installing the latest one

              Install BigWig Artifact: BigWig uses the htsjdk library. Since IGB depends on BigWig, the htsjdk version in BigWig needs to be upgraded to 3.0.0 to avoid dependency conflicts.

            • Clone BigWig into your local.
            • In build.gradle file, change htsjdk-igb version to 3.0.0
              • May also need to add mavenLocal() under repositories in the build.gradle file
            • Run: gradle build -x test
            • Install the artifact to your local maven repository using the below command (make sure you delete the existing bigwig artifact before installing the below one):
              Installing BigWig artifact

              mvn install:install-file -Dfile="<path>/bigwig-2.0.0.jar" -DgroupId="org.broad.igv" -DartifactId="bigwig" -Dversion="2.0.0" -Dpackaging="jar" -DgeneratePom="true"


            Testing IGB:

            Show
            kgopu Kaushik Gopu added a comment - - edited Testing Instructions: Clone this repository into your local and checkout to branch igb_main-3.0.0 . Run: command gradle clean build -x test A folder named build will appear after running the above command. Look for htsjdk-igb-3.0.0.jar in build/libs folder. Install the artifact in your local maven repository using the below command: Installing artifact locally mvn install:install-file -Dfile="<path to jar file>" -DgroupId="htsjdk-igb" -DartifactId="htsjdk-igb" -Dversion="3.0.0" -Dpackaging="jar" -DgeneratePom="true" Make sure you delete the old version of htsjdk before installing the latest one Install BigWig Artifact: BigWig uses the htsjdk library. Since IGB depends on BigWig, the htsjdk version in BigWig needs to be upgraded to 3.0.0 to avoid dependency conflicts. Clone BigWig into your local. In build.gradle file, change htsjdk-igb version to 3.0.0 May also need to add mavenLocal() under repositories in the build.gradle file Run: gradle build -x test Install the artifact to your local maven repository using the below command (make sure you delete the existing bigwig artifact before installing the below one): Installing BigWig artifact mvn install:install-file -Dfile="<path>/bigwig-2.0.0.jar" -DgroupId="org.broad.igv" -DartifactId="bigwig" -Dversion="2.0.0" -Dpackaging="jar" -DgeneratePom="true" Testing IGB: Switch to branch: https://bitbucket.org/kaushik-gopu/kgopu_integrated-genome-browser/branch/IGBF-3464 Run mvn clean install Test IGB by inputting different track formats such as BAI, BED, BAM and also keep monitoring logs while testing.
            kgopu Kaushik Gopu made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            nfreese Nowlan Freese made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            nfreese Nowlan Freese made changes -
            Assignee Kaushik Gopu [ kgopu ] Nowlan Freese [ nfreese ]
            Hide
            nfreese Nowlan Freese added a comment - - edited

            Tested following above instructions on Mac:

            • Able to add and load data from a bam file from IGB Quickload.
            • Able to add and load data (visualize) a bai file.
            • Able to create a depth graph from a bam file.
            • Able to add and load data from a local bam file.
            • Able to add and load data from a local bed file.
            • Able to add and load data from a local sam file.

            Kaushik Gopu - everything seems to be working as expected that I can think of. Please move forward with pull requests once artifacts have been deployed.

            Show
            nfreese Nowlan Freese added a comment - - edited Tested following above instructions on Mac: Able to add and load data from a bam file from IGB Quickload. Able to add and load data (visualize) a bai file. Able to create a depth graph from a bam file. Able to add and load data from a local bam file. Able to add and load data from a local bed file. Able to add and load data from a local sam file. Kaushik Gopu - everything seems to be working as expected that I can think of. Please move forward with pull requests once artifacts have been deployed.
            nfreese Nowlan Freese made changes -
            Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ] Kaushik Gopu [ kgopu ]
            nfreese Nowlan Freese made changes -
            Link This issue relates to IGBF-3484 [ IGBF-3484 ]
            Hide
            ann.loraine Ann Loraine added a comment -

            The htsjdk artifact can't be deployed just yet because the bundle symbolic name is wrong. It's showing up as "htsjdk" in the MANIFEST but it should instead be "htsjdk-igb"

            Moving back to "To-Do"

            Show
            ann.loraine Ann Loraine added a comment - The htsjdk artifact can't be deployed just yet because the bundle symbolic name is wrong. It's showing up as "htsjdk" in the MANIFEST but it should instead be "htsjdk-igb" Moving back to "To-Do"
            ann.loraine Ann Loraine made changes -
            Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
            ann.loraine Ann Loraine made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            ann.loraine Ann Loraine made changes -
            Status Reviewing Pull Request [ 10303 ] To-Do [ 10305 ]
            ann.loraine Ann Loraine made changes -
            Assignee Kaushik Gopu [ kgopu ]
            nfreese Nowlan Freese made changes -
            Description Situation: In order to add CRAM file support (IGBF-1469) we need to update the HTSJDK dependency version. Currently we are using the artifact htsjdk-igb.

            Task: Investigate our use of the htsjd-igb dependency. Determine how we can update this dependency.

            Github repo: https://github.com/lorainelab/htsjdk-igb
            Situation: In order to add CRAM file support (IGBF-1469) we need to update the HTSJDK dependency version. Currently we are using the artifact htsjdk-igb.

            Task: Investigate our use of the htsjdk-igb dependency. Determine how we can update this dependency.

            Github repo: https://github.com/lorainelab/htsjdk-igb
            nfreese Nowlan Freese made changes -
            Summary Investigate updating HTSJDK Investigate updating HTSJDK and deploying the artifact
            nfreese Nowlan Freese made changes -
            Summary Investigate updating HTSJDK and deploying the artifact Investigate updating/deploying htsjdk-igb
            ann.loraine Ann Loraine made changes -
            Sprint Fall 4 2023 Oct 16, Fall 5 [ 180, 181 ] Fall 4 2023 Oct 16, Fall 5, Fall 6 [ 180, 181, 182 ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            nfreese Nowlan Freese made changes -
            Assignee Ann Loraine [ aloraine ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ]
            nfreese Nowlan Freese made changes -
            Assignee Ann Loraine [ aloraine ]
            ann.loraine Ann Loraine made changes -
            Summary Investigate updating/deploying htsjdk-igb Deploy htsjdk-igb
            ann.loraine Ann Loraine made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            ann.loraine Ann Loraine added a comment -

            Bundle, javadoc and source file all deployed and ready for testing.

            See: https://nexus.bioviz.org/#browse/browse:maven-releases:htsjdk-igb

            Show
            ann.loraine Ann Loraine added a comment - Bundle, javadoc and source file all deployed and ready for testing. See: https://nexus.bioviz.org/#browse/browse:maven-releases:htsjdk-igb
            ann.loraine Ann Loraine made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            ann.loraine Ann Loraine made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            ann.loraine Ann Loraine made changes -
            Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
            ann.loraine Ann Loraine made changes -
            Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
            ann.loraine Ann Loraine made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            ann.loraine Ann Loraine made changes -
            Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ]
            nfreese Nowlan Freese made changes -
            Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ]
            Hide
            nfreese Nowlan Freese added a comment -

            Bundle appears to be deployed successfully. The bundle is successfully retrieved and installed in my local .m2 when building IGB and specifying htsjdk-igb version 3.0.5.

            Closing ticket.

            Show
            nfreese Nowlan Freese added a comment - Bundle appears to be deployed successfully. The bundle is successfully retrieved and installed in my local .m2 when building IGB and specifying htsjdk-igb version 3.0.5. Closing ticket.
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ] Kaushik Gopu [ kgopu ]
            nfreese Nowlan Freese made changes -
            Resolution Done [ 10000 ]
            Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]
            Hide
            kgopu Kaushik Gopu added a comment - - edited

            Updated IGB's htsjdk and bigWig versions to 3.0.5 and 2.0.1, respectively in the IGB project pom.
            Changes are available to in this commit
            Moving to first level review

            Show
            kgopu Kaushik Gopu added a comment - - edited Updated IGB's htsjdk and bigWig versions to 3.0.5 and 2.0.1, respectively in the IGB project pom. Changes are available to in this commit Moving to first level review
            kgopu Kaushik Gopu made changes -
            Resolution Done [ 10000 ]
            Status Closed [ 6 ] To-Do [ 10305 ]
            kgopu Kaushik Gopu made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            kgopu Kaushik Gopu made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            kgopu Kaushik Gopu made changes -
            Assignee Kaushik Gopu [ kgopu ] Nowlan Freese [ nfreese ]
            nfreese Nowlan Freese made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            Hide
            nfreese Nowlan Freese added a comment -

            Commit looks good, ready for pull request.

            Note that it would be best to deploy the changes to Bigwig (see IGBF-3465) prior to merging this pull request.

            Show
            nfreese Nowlan Freese added a comment - Commit looks good, ready for pull request. Note that it would be best to deploy the changes to Bigwig (see IGBF-3465 ) prior to merging this pull request.
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ] Kaushik Gopu [ kgopu ]
            nfreese Nowlan Freese made changes -
            Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
            Hide
            kgopu Kaushik Gopu added a comment -
            • PR has been submitted.
            • It would be best to deploy bigWig (IGBF-3464) prior to merging this PR
            Show
            kgopu Kaushik Gopu added a comment - PR has been submitted. It would be best to deploy bigWig ( IGBF-3464 ) prior to merging this PR
            kgopu Kaushik Gopu made changes -
            Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
            kgopu Kaushik Gopu made changes -
            Assignee Kaushik Gopu [ kgopu ] Ann Loraine [ aloraine ]
            ann.loraine Ann Loraine made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            ann.loraine Ann Loraine made changes -
            Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ]
            Hide
            ann.loraine Ann Loraine added a comment -

            PR is merged and IGB is built, with no errors.

            attn: Kaushik Gopu

            Show
            ann.loraine Ann Loraine added a comment - PR is merged and IGB is built, with no errors. attn: Kaushik Gopu
            ann.loraine Ann Loraine made changes -
            Sprint Fall 4 2023 Oct 16, Fall 5, Fall 6 [ 180, 181, 182 ] Fall 4 2023 Oct 16, Fall 5, Fall 6, Fall 7 [ 180, 181, 182, 183 ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            Hide
            nfreese Nowlan Freese added a comment -

            Tested on Mac using strategy described in one of my previous comments. Everything is working correctly, closing ticket.

            Show
            nfreese Nowlan Freese added a comment - Tested on Mac using strategy described in one of my previous comments. Everything is working correctly, closing ticket.
            nfreese Nowlan Freese made changes -
            Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
            nfreese Nowlan Freese made changes -
            Resolution Done [ 10000 ]
            Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]
            nfreese Nowlan Freese made changes -
            Assignee Kaushik Gopu [ kgopu ]
            pkulzer Paige Kulzer made changes -
            Fix Version/s 10.0.0 Major Release [ 10900 ]

              People

              • Assignee:
                kgopu Kaushik Gopu
                Reporter:
                nfreese Nowlan Freese
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: