Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-1767

Fix bundle resolution error involving htsjdk-igb

    Details

    • Story Points:
      2
    • Sprint:
      Summer 2019 Sprint 8, Summer 2019 Sprint 9, Summer 2019 Sprint 10, Summer 2019 Sprint 11

      Description

      For details see:

      Investigate bundle resolution error involving htsjdk-igb
      https://jira.transvar.org/browse/IGBF-1627

      This requires understanding OSGi and how bundle dependencies get resolved during bundle installation and activation.

      Also, review how we made htsjdk-igb bundle in the first place by looking at linked Jira issues.

      Probably we have to make some changes to how we generate that bundle?

      Potentially useful resources:

      Please add more as you work on this.

      When done, please write up a short, plain English description of what you did to fix the problem. (This can be a useful case study to introduce devs to OSGi in IGB.)
      *

        Attachments

          Issue Links

            Activity

            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment -

            [~aloraine]
            Investigation Results:

            htsjdk requires below jar to remove bundle resolution error. I manually downloaded this jar and copied them in /bundles.
            1. commons-jexl-2.1.1.jar
            jexl requires below jar to remove bundle resolution error. I manually downloaded this jar and copied them in /bundles.
            1. commons-logging-1.2.jar

            I manually copied these two jars in /bundles to remove bundle resolution error and now IGB runs without any error. These two jars should be downloaded from pom.xml (Maven). But for some reason maven is not downloading these jars even after providing dependencies.

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - [~aloraine] Investigation Results: htsjdk requires below jar to remove bundle resolution error. I manually downloaded this jar and copied them in /bundles. 1. commons-jexl-2.1.1.jar jexl requires below jar to remove bundle resolution error. I manually downloaded this jar and copied them in /bundles. 1. commons-logging-1.2.jar I manually copied these two jars in /bundles to remove bundle resolution error and now IGB runs without any error. These two jars should be downloaded from pom.xml (Maven). But for some reason maven is not downloading these jars even after providing dependencies.
            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - - edited

            Steps I followed to find the missing jars to remove bundle resolution error in htsjdk. (USEFUL)

            Note: Manually copying these jars in IGB can fix this error. But we need to download it as a maven dependency. Currently, IGB is not downloading these jars even after providing these dependencies in pom.xml. (FIXED IT- Check below comment)

            1. Download Felix and setup web console for Felix. Follow this tutorial for setting up Felix in your localhost. link: https://www.youtube.com/watch?v=KqhpgiFNQso&list=PLlf_Fk4pk-gigRWeoiSw0lsLyx97ICzQb
            2. http://localhost:8080/system/console/bundles - load this link to enter Felix web console. (Alternative to Felix GOGO shell)
            3. Now install jar which needs a fix. In my case, htsjdk is having a bundle resolution error. So I downloaded htsjdk jar.
            4. Use web console to install and start bundle. You can also use Felix GoGo Shell for doing this but Felix web console is more user-friendly.
            5. I have installed htsjdk bundle but unable to start it. Expand htsjdk to see missing requirements. Missing requirements are marked in RED COLOUR as shown in attached image MissingRequirements.jpg.
            6. I have installed and started commons-compress-1.5.jar file to remove compress Requirement. Check MissingRequirements_afterInstallingCompress.jpg for the more missing requirement.
            Note: 7. Reload the page after installing each bundle to get the updated list.
            7. Now, htsjdk is missing commons-jexl-2.1.1.jar. I downloaded this jar, installed it and started. Now all the htsjdk requirements are satisfied but both htsjdk and jexl bundles are not starting.
            8. I have expanded jexl to get more information and found out that jexl requirement is not satisfied. jexl missing requirement is shown in jexl_Missing_Requirement.JPG attachment.
            9. jexl requires logging bundle to start. Downloaded commons-logging-1.2.jar, installed it and started the bundle.
            10. Now, I could able to start htsjdk without any bundle resolution error.

            We require three jars to start htsjdk bundle without bundle resolution error, they are:
            1. commons-compress-1.5.jar (This jar is already available in IGB. We dont have to fix this)
            2. commons-jexl-2.1.1.jar
            3. commons-logging-1.2.jar

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - - edited Steps I followed to find the missing jars to remove bundle resolution error in htsjdk. (USEFUL) Note: Manually copying these jars in IGB can fix this error. But we need to download it as a maven dependency. Currently, IGB is not downloading these jars even after providing these dependencies in pom.xml. (FIXED IT- Check below comment) 1. Download Felix and setup web console for Felix. Follow this tutorial for setting up Felix in your localhost. link: https://www.youtube.com/watch?v=KqhpgiFNQso&list=PLlf_Fk4pk-gigRWeoiSw0lsLyx97ICzQb 2. http://localhost:8080/system/console/bundles - load this link to enter Felix web console. (Alternative to Felix GOGO shell) 3. Now install jar which needs a fix. In my case, htsjdk is having a bundle resolution error. So I downloaded htsjdk jar. 4. Use web console to install and start bundle. You can also use Felix GoGo Shell for doing this but Felix web console is more user-friendly. 5. I have installed htsjdk bundle but unable to start it. Expand htsjdk to see missing requirements. Missing requirements are marked in RED COLOUR as shown in attached image MissingRequirements.jpg. 6. I have installed and started commons-compress-1.5.jar file to remove compress Requirement. Check MissingRequirements_afterInstallingCompress.jpg for the more missing requirement. Note: 7. Reload the page after installing each bundle to get the updated list. 7. Now, htsjdk is missing commons-jexl-2.1.1.jar. I downloaded this jar, installed it and started. Now all the htsjdk requirements are satisfied but both htsjdk and jexl bundles are not starting. 8. I have expanded jexl to get more information and found out that jexl requirement is not satisfied. jexl missing requirement is shown in jexl_Missing_Requirement.JPG attachment. 9. jexl requires logging bundle to start. Downloaded commons-logging-1.2.jar, installed it and started the bundle. 10. Now, I could able to start htsjdk without any bundle resolution error. We require three jars to start htsjdk bundle without bundle resolution error, they are: 1. commons-compress-1.5.jar (This jar is already available in IGB. We dont have to fix this) 2. commons-jexl-2.1.1.jar 3. commons-logging-1.2.jar
            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - - edited

            [~aloraine]
            Branch: https://bitbucket.org/svallapu/charan_igb/branch/IGBF-1767 (DONT USE.. USE BELOW)

            htsjdk module is successfully started without bundle resolution error. Check htsjdk_success.JPG attachment for more information.

            Fix:

            All the apache.commons dependencies are coming from shared-lib-wrapper module.

            shared-lib-wrapper module pom.xml is missing above mentioned two dependencies (jexl and logging). Adding these dependencies to the pom.xml solved the problem.

            Added below artefacts.
            <artifactItem>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
            <version>$

            {commonsLoggingVersion}

            </version>
            </artifactItem>
            <artifactItem>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-jexl</artifactId>
            <version>$

            {commonsJexl}

            </version>
            </artifactItem>

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - - edited [~aloraine] Branch: https://bitbucket.org/svallapu/charan_igb/branch/IGBF-1767 (DONT USE.. USE BELOW) htsjdk module is successfully started without bundle resolution error. Check htsjdk_success.JPG attachment for more information. Fix: All the apache.commons dependencies are coming from shared-lib-wrapper module. shared-lib-wrapper module pom.xml is missing above mentioned two dependencies (jexl and logging). Adding these dependencies to the pom.xml solved the problem. Added below artefacts. <artifactItem> <groupId>commons-logging</groupId> <artifactId>commons-logging</artifactId> <version>$ {commonsLoggingVersion} </version> </artifactItem> <artifactItem> <groupId>org.apache.commons</groupId> <artifactId>commons-jexl</artifactId> <version>$ {commonsJexl} </version> </artifactItem>
            Hide
            ann.loraine Ann Loraine added a comment -

            Thanks for these incredibly detailed – and useful! – notes!
            I have a question:
            Can the two missing dependencies be treated as optional dependencies for htsjdk library?

            I ask because it seems like the functionality we need from the htsjdk – BAM, SAM, and tabix format parsing – is working OK once the library is loaded into the IGB application. That is, the framework seems to be able to make the key packages available to IGB. Or am I misunderstanding?

            If this is correct, then we probably do not need to package these additional jars into the IGB installer. Instead, we could modify the POM and MANIFEST for the htsjdk bundle to declare the to missing dependencies as optional.

            Some documentation for how this can be done is available from our old friend the BND tool here:
            https://bnd.bndtools.org/chapters/390-wrapping.html

            If at all possible, I would like to avoid adding more stuff to the IGB installer. But if you think we should for sure include these two libraries, I would be cool with it.

            Please take a look at the above link and add some comments with your thoughts.

            Show
            ann.loraine Ann Loraine added a comment - Thanks for these incredibly detailed – and useful! – notes! I have a question: Can the two missing dependencies be treated as optional dependencies for htsjdk library? I ask because it seems like the functionality we need from the htsjdk – BAM, SAM, and tabix format parsing – is working OK once the library is loaded into the IGB application. That is, the framework seems to be able to make the key packages available to IGB. Or am I misunderstanding? If this is correct, then we probably do not need to package these additional jars into the IGB installer. Instead, we could modify the POM and MANIFEST for the htsjdk bundle to declare the to missing dependencies as optional. Some documentation for how this can be done is available from our old friend the BND tool here: https://bnd.bndtools.org/chapters/390-wrapping.html If at all possible, I would like to avoid adding more stuff to the IGB installer. But if you think we should for sure include these two libraries, I would be cool with it. Please take a look at the above link and add some comments with your thoughts.
            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment -

            [~aloraine]

            htsjdk is downloaded as a maven dependency where ever required. We don't have to create it as an OSGi bundle.

            for example: In genometry bundle htsjdk requirement is satisfied as a maven dependency. (pom.xml[genometry])

            I have removed htsjdk in shared lib wrapper. By removing htsjdk in shared lib wrapper, IGB won't create a htsjdk.jar file in /bundles. If there is no htsjdk.jar file in /bundles, IGB won't create an OSGi bundle.

            Branch: https://bitbucket.org/svallapu/charan_igb/branch/IGBF-1767-latest

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - [~aloraine] htsjdk is downloaded as a maven dependency where ever required. We don't have to create it as an OSGi bundle. for example: In genometry bundle htsjdk requirement is satisfied as a maven dependency. (pom.xml [genometry] ) I have removed htsjdk in shared lib wrapper. By removing htsjdk in shared lib wrapper, IGB won't create a htsjdk.jar file in /bundles. If there is no htsjdk.jar file in /bundles, IGB won't create an OSGi bundle. Branch: https://bitbucket.org/svallapu/charan_igb/branch/IGBF-1767-latest
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Genometry uses packages from htsjdk-igb. I think it is the only part of the code base that uses these packages. The pom.xml for genometry declares that htsdjk-igb is a dependency that is being provided by the framework:
            snip
            <dependency>
            <groupId>htsjdk-igb</groupId>
            <artifactId>htsjdk-igb</artifactId>
            <scope>provided</scope>
            </dependency>
            snip

            Show
            ann.loraine Ann Loraine added a comment - - edited Genometry uses packages from htsjdk-igb. I think it is the only part of the code base that uses these packages. The pom.xml for genometry declares that htsdjk-igb is a dependency that is being provided by the framework: snip <dependency> <groupId>htsjdk-igb</groupId> <artifactId>htsjdk-igb</artifactId> <scope>provided</scope> </dependency> snip
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            We should probably indicate which version of htsjdk-igb that the genometry bundle depends on.
            We can do this by adding:

            snip
            <version>
            $

            {htsjdk-igb-Version}

            </version>
            snip

            to the above. This variable is defined in the parent POM.

            Show
            ann.loraine Ann Loraine added a comment - - edited We should probably indicate which version of htsjdk-igb that the genometry bundle depends on. We can do this by adding: snip <version> $ {htsjdk-igb-Version} </version> snip to the above. This variable is defined in the parent POM.
            Hide
            ann.loraine Ann Loraine added a comment -

            Source code for htsjdk-igb version 2.16.2: https://github.com/lorainelab/htsjdk/tree/igb-2.16.2

            Show
            ann.loraine Ann Loraine added a comment - Source code for htsjdk-igb version 2.16.2: https://github.com/lorainelab/htsjdk/tree/igb-2.16.2
            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - - edited

            [~aloraine]

            Branch: https://bitbucket.org/svallapu/charan_igb/branch/IGBF-1767#diff

            htsjdk-igb is exported from genometry. This is how IGB gets htsjdk-igb, but IGB should get it from htsjdk-igb OSGi bundle.

            Quickload and das are getting htsjdk-igb from genometry. It should get it from htsjdk-igb OSGi bundle.

            FIX:

            1. Remove htsjdk-igb from Export Packages in genometry bundle pom.xml.
            2. To make htsjdk-igb bundle active, add jexl and logging in shared lib wrappers. [These two are OSGi bundles]. We have to start these two as OSGi bundles.
            3. Add htsjdk-igb bundle in quickload and das.

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - - edited [~aloraine] Branch: https://bitbucket.org/svallapu/charan_igb/branch/IGBF-1767#diff htsjdk-igb is exported from genometry. This is how IGB gets htsjdk-igb, but IGB should get it from htsjdk-igb OSGi bundle. Quickload and das are getting htsjdk-igb from genometry. It should get it from htsjdk-igb OSGi bundle. FIX: 1. Remove htsjdk-igb from Export Packages in genometry bundle pom.xml. 2. To make htsjdk-igb bundle active, add jexl and logging in shared lib wrappers. [These two are OSGi bundles] . We have to start these two as OSGi bundles. 3. Add htsjdk-igb bundle in quickload and das.
            Hide
            ann.loraine Ann Loraine added a comment -

            I agree this is the right solution.

            Show
            ann.loraine Ann Loraine added a comment - I agree this is the right solution.
            Hide
            ann.loraine Ann Loraine added a comment -

            Used bnd command line tool to look into the MANIFEST.MF of htsdjk-igb 2.16.2 being downloaded by IGB during a build process:

            • bnd print htsjdk-igb-2.16.2.jar

            Output (attached) indicates the bundle is exporting all packages as version 2.16.2 (which is fine) but is also importing packages from the same library. I don't think this is necessary and probably could lead to bad bugs. Red Hat summarizes conventions for what types of bundles should import their own exported packages - see table in: https://access.redhat.com/documentation/en-US/Fuse_ESB/4.4.1/html-single/Managing_OSGi_Dependencies/index.html#BCL

            The bundle MANIFEST for htsdjk-igb (as expected) includes import statements from non-htsjdk packages:

            • snip *
              javax.script
              javax.xml.bind
              javax.xml.bind.annotation
              org.apache.commons.compress.compressors.bzip2 {version=[1.4,2)}
              org.apache.commons.compress.compressors.xz {version=[1.4,2)}

              org.apache.commons.compress.utils

              {version=[1.4,2)}

              org.apache.commons.jexl2

              {version=[2.1,3)}
            • snp *

            As noted in previous comments, commons-compress-1.5.jar is available already in the IGB run-time. But commons-jexl-2.1.1.jar and commons-logging-1.2.jar are not.

            I have a doubt:

            I am not sure if the parts of htsdjk-igb that are actually being used in IGB are needing to access functionality of the commons-jexl-2.1.1.jar bundle (and its declared requirement bundle commons-logging-1.2.jar). If yes, this means that we could declare packages from commons-jexl-2.1.1.jar as optional in the MANIFEST of htsjkd-igb.

            That is, we could change the import statement for org.apache.commons.jexl2 to look like:

            • org.apache.commons.jexl2;resolution:=optional;version="[2.1,3)"

            instead of

            • org.apache.commons.jexl2;version="[2.1,3)"

            In addition, we would want to no longer export this same library from htsdjk-igb bundle.

            The htsdjk-igb bundle MANIFEST can be re-built by modifying build.gradle in our repository https://github.com/lorainelab/htsjdk/tree/igb-2.16.2 (a branch)

            Or we could daringly edit it by hand and see what happens.

            Suggestion:

            • Clone the repo
            • Investigate options for osgi plug-in being used to build the bundle (to avoid building by hand if too inconvenient)
            • For testing, build the bundle (jar file) and check syntax of MANIFEST using command line bnd as above
            Show
            ann.loraine Ann Loraine added a comment - Used bnd command line tool to look into the MANIFEST.MF of htsdjk-igb 2.16.2 being downloaded by IGB during a build process: bnd print htsjdk-igb-2.16.2.jar Output (attached) indicates the bundle is exporting all packages as version 2.16.2 (which is fine) but is also importing packages from the same library. I don't think this is necessary and probably could lead to bad bugs. Red Hat summarizes conventions for what types of bundles should import their own exported packages - see table in: https://access.redhat.com/documentation/en-US/Fuse_ESB/4.4.1/html-single/Managing_OSGi_Dependencies/index.html#BCL The bundle MANIFEST for htsdjk-igb (as expected) includes import statements from non-htsjdk packages: snip * javax.script javax.xml.bind javax.xml.bind.annotation org.apache.commons.compress.compressors.bzip2 {version=[1.4,2)} org.apache.commons.compress.compressors.xz {version=[1.4,2)} org.apache.commons.compress.utils {version=[1.4,2)} org.apache.commons.jexl2 {version=[2.1,3)} snp * As noted in previous comments, commons-compress-1.5.jar is available already in the IGB run-time. But commons-jexl-2.1.1.jar and commons-logging-1.2.jar are not. I have a doubt: I am not sure if the parts of htsdjk-igb that are actually being used in IGB are needing to access functionality of the commons-jexl-2.1.1.jar bundle (and its declared requirement bundle commons-logging-1.2.jar). If yes, this means that we could declare packages from commons-jexl-2.1.1.jar as optional in the MANIFEST of htsjkd-igb. That is, we could change the import statement for org.apache.commons.jexl2 to look like: org.apache.commons.jexl2;resolution:=optional;version="[2.1,3)" instead of org.apache.commons.jexl2;version="[2.1,3)" In addition, we would want to no longer export this same library from htsdjk-igb bundle. The htsdjk-igb bundle MANIFEST can be re-built by modifying build.gradle in our repository https://github.com/lorainelab/htsjdk/tree/igb-2.16.2 (a branch) Or we could daringly edit it by hand and see what happens. Suggestion: Clone the repo Investigate options for osgi plug-in being used to build the bundle (to avoid building by hand if too inconvenient) For testing, build the bundle (jar file) and check syntax of MANIFEST using command line bnd as above
            Hide
            ann.loraine Ann Loraine added a comment -

            I used

            • bnd print -u htsjdk-igb-2.16.2.jar

            to identify packages within htsjdk that are using org.apache.commons.jexl2 artifact.

            The only package in htsjdk that uses org.apache.commons.jexl2 is:

            • htsjdk.variant.variantcontext
            Show
            ann.loraine Ann Loraine added a comment - I used bnd print -u htsjdk-igb-2.16.2.jar to identify packages within htsjdk that are using org.apache.commons.jexl2 artifact. The only package in htsjdk that uses org.apache.commons.jexl2 is: htsjdk.variant.variantcontext
            Hide
            ann.loraine Ann Loraine added a comment -

            Our build.gradle file is using plugin osgi.

            According to:
            https://github.com/bndtools/bnd/blob/master/biz.aQute.bnd.gradle/README.md#gradle-plugin-for-workspace-builds

            we should biz.aQute.bnd.builder plugin instead.

            The above link contains example syntax for adding stuff to the MANIFEST.

            For example:

            jar {
            manifest

            { attributes('Export-Package': 'com.acme.api.*', '-sources': 'true', '-include': 'other.bnd') }

            }

            Show
            ann.loraine Ann Loraine added a comment - Our build.gradle file is using plugin osgi. According to: https://github.com/bndtools/bnd/blob/master/biz.aQute.bnd.gradle/README.md#gradle-plugin-for-workspace-builds we should biz.aQute.bnd.builder plugin instead. The above link contains example syntax for adding stuff to the MANIFEST. For example: jar { manifest { attributes('Export-Package': 'com.acme.api.*', '-sources': 'true', '-include': 'other.bnd') } }
            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - - edited

            [~aloraine]
            According to my understanding jexl2 is used by htsjdk-igb (htsjdk.variant.variantcontext). Does it mean we cannot keep jexl2 as optional or am I wrong?

            htsjdk.variant.variantcontext is extensively using jexl.
            jexl is used in below java files.
            1. src\main\java\htsjadk\variant\varaintcontext\JEXLMap.java
            2. src\main\java\htsjadk\variant\varaintcontext\JexlMissingValueTreatment.java
            3. src\main\java\htsjadk\variant\varaintcontext\VariantContextUtils.java
            4. src\main\java\htsjadk\variant\varaintcontext\VaraintJEXLContext.java
            5. src\main\java\htsjadk\variant\varaintcontext\VaraintJEXLContextUnitTest.java
            6. src\main\java\htsjadk\variant\varaintcontext\GenotypeJEXLContext.java

            Now the issue is htsjdk is exporting (2.16.2) and importing(2.16.3) the same library. I will look into it.

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - - edited [~aloraine] According to my understanding jexl2 is used by htsjdk-igb (htsjdk.variant.variantcontext). Does it mean we cannot keep jexl2 as optional or am I wrong? htsjdk.variant.variantcontext is extensively using jexl. jexl is used in below java files. 1. src\main\java\htsjadk\variant\varaintcontext\JEXLMap.java 2. src\main\java\htsjadk\variant\varaintcontext\JexlMissingValueTreatment.java 3. src\main\java\htsjadk\variant\varaintcontext\VariantContextUtils.java 4. src\main\java\htsjadk\variant\varaintcontext\VaraintJEXLContext.java 5. src\main\java\htsjadk\variant\varaintcontext\VaraintJEXLContextUnitTest.java 6. src\main\java\htsjadk\variant\varaintcontext\GenotypeJEXLContext.java Now the issue is htsjdk is exporting (2.16.2) and importing(2.16.3) the same library. I will look into it.
            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - Branch: https://bitbucket.org/svallapu/charan_igb/branch/IGBF-1767#diff
            Hide
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment -

            Pull request submitted.

            Show
            svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - Pull request submitted.
            Hide
            noor91zahara Noor Zahara (Inactive) added a comment -

            Works as expected.

            Show
            noor91zahara Noor Zahara (Inactive) added a comment - Works as expected.

              People

              • Assignee:
                svallapu Sai Charan Reddy Vallapureddy (Inactive)
                Reporter:
                ann.loraine Ann Loraine
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: