[~aloraine]
Investigation Results:

htsjdk requires below jar to remove bundle resolution error. I manually downloaded this jar and copied them in /bundles.
1. commons-jexl-2.1.1.jar
jexl requires below jar to remove bundle resolution error. I manually downloaded this jar and copied them in /bundles.
1. commons-logging-1.2.jar

I manually copied these two jars in /bundles to remove bundle resolution error and now IGB runs without any error. These two jars should be downloaded from pom.xml (Maven). But for some reason maven is not downloading these jars even after providing dependencies.

Steps I followed to find the missing jars to remove bundle resolution error in htsjdk. (USEFUL)

Note: Manually copying these jars in IGB can fix this error. But we need to download it as a maven dependency. Currently, IGB is not downloading these jars even after providing these dependencies in pom.xml. (FIXED IT- Check below comment)

1. Download Felix and setup web console for Felix. Follow this tutorial for setting up Felix in your localhost. link: https://www.youtube.com/watch?v=KqhpgiFNQso&list=PLlf_Fk4pk-gigRWeoiSw0lsLyx97ICzQb
2. http://localhost:8080/system/console/bundles - load this link to enter Felix web console. (Alternative to Felix GOGO shell)
3. Now install jar which needs a fix. In my case, htsjdk is having a bundle resolution error. So I downloaded htsjdk jar.
4. Use web console to install and start bundle. You can also use Felix GoGo Shell for doing this but Felix web console is more user-friendly.
5. I have installed htsjdk bundle but unable to start it. Expand htsjdk to see missing requirements. Missing requirements are marked in RED COLOUR as shown in attached image MissingRequirements.jpg.
6. I have installed and started commons-compress-1.5.jar file to remove compress Requirement. Check MissingRequirements_afterInstallingCompress.jpg for the more missing requirement.
Note: 7. Reload the page after installing each bundle to get the updated list.
7. Now, htsjdk is missing commons-jexl-2.1.1.jar. I downloaded this jar, installed it and started. Now all the htsjdk requirements are satisfied but both htsjdk and jexl bundles are not starting.
8. I have expanded jexl to get more information and found out that jexl requirement is not satisfied. jexl missing requirement is shown in jexl_Missing_Requirement.JPG attachment.
9. jexl requires logging bundle to start. Downloaded commons-logging-1.2.jar, installed it and started the bundle.
10. Now, I could able to start htsjdk without any bundle resolution error.

We require three jars to start htsjdk bundle without bundle resolution error, they are:
1. commons-compress-1.5.jar (This jar is already available in IGB. We dont have to fix this)
2. commons-jexl-2.1.1.jar
3. commons-logging-1.2.jar

[~aloraine]
Branch: https://bitbucket.org/svallapu/charan_igb/branch/IGBF-1767 (DONT USE.. USE BELOW)

htsjdk module is successfully started without bundle resolution error. Check htsjdk_success.JPG attachment for more information.

Fix:

All the apache.commons dependencies are coming from shared-lib-wrapper module.

shared-lib-wrapper module pom.xml is missing above mentioned two dependencies (jexl and logging). Adding these dependencies to the pom.xml solved the problem.

Added below artefacts.
<artifactItem>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>$

</version>
</artifactItem>
<artifactItem>
<groupId>org.apache.commons</groupId>
<artifactId>commons-jexl</artifactId>
<version>$

</version>
</artifactItem>

Thanks for these incredibly detailed – and useful! – notes!
I have a question:
Can the two missing dependencies be treated as optional dependencies for htsjdk library?

I ask because it seems like the functionality we need from the htsjdk – BAM, SAM, and tabix format parsing – is working OK once the library is loaded into the IGB application. That is, the framework seems to be able to make the key packages available to IGB. Or am I misunderstanding?

If this is correct, then we probably do not need to package these additional jars into the IGB installer. Instead, we could modify the POM and MANIFEST for the htsjdk bundle to declare the to missing dependencies as optional.

Some documentation for how this can be done is available from our old friend the BND tool here:
https://bnd.bndtools.org/chapters/390-wrapping.html

If at all possible, I would like to avoid adding more stuff to the IGB installer. But if you think we should for sure include these two libraries, I would be cool with it.

Please take a look at the above link and add some comments with your thoughts.

[~aloraine]

htsjdk is downloaded as a maven dependency where ever required. We don't have to create it as an OSGi bundle.

for example: In genometry bundle htsjdk requirement is satisfied as a maven dependency. (pom.xml[genometry])

I have removed htsjdk in shared lib wrapper. By removing htsjdk in shared lib wrapper, IGB won't create a htsjdk.jar file in /bundles. If there is no htsjdk.jar file in /bundles, IGB won't create an OSGi bundle.

Branch: https://bitbucket.org/svallapu/charan_igb/branch/IGBF-1767-latest

Genometry uses packages from htsjdk-igb. I think it is the only part of the code base that uses these packages. The pom.xml for genometry declares that htsdjk-igb is a dependency that is being provided by the framework:
snip
<dependency>
<groupId>htsjdk-igb</groupId>
<artifactId>htsjdk-igb</artifactId>
<scope>provided</scope>
</dependency>
snip

We should probably indicate which version of htsjdk-igb that the genometry bundle depends on.
We can do this by adding:

snip
<version>
$

</version>
snip

to the above. This variable is defined in the parent POM.

Source code for htsjdk-igb version 2.16.2: https://github.com/lorainelab/htsjdk/tree/igb-2.16.2

[~aloraine]

Branch: https://bitbucket.org/svallapu/charan_igb/branch/IGBF-1767#diff

htsjdk-igb is exported from genometry. This is how IGB gets htsjdk-igb, but IGB should get it from htsjdk-igb OSGi bundle.

Quickload and das are getting htsjdk-igb from genometry. It should get it from htsjdk-igb OSGi bundle.

FIX:

1. Remove htsjdk-igb from Export Packages in genometry bundle pom.xml.
2. To make htsjdk-igb bundle active, add jexl and logging in shared lib wrappers. [These two are OSGi bundles]. We have to start these two as OSGi bundles.
3. Add htsjdk-igb bundle in quickload and das.

I agree this is the right solution.

Used bnd command line tool to look into the MANIFEST.MF of htsdjk-igb 2.16.2 being downloaded by IGB during a build process:

• bnd print htsjdk-igb-2.16.2.jar

Output (attached) indicates the bundle is exporting all packages as version 2.16.2 (which is fine) but is also importing packages from the same library. I don't think this is necessary and probably could lead to bad bugs. Red Hat summarizes conventions for what types of bundles should import their own exported packages - see table in: https://access.redhat.com/documentation/en-US/Fuse_ESB/4.4.1/html-single/Managing_OSGi_Dependencies/index.html#BCL

The bundle MANIFEST for htsdjk-igb (as expected) includes import statements from non-htsjdk packages:

• snip *
  javax.script
  javax.xml.bind
  javax.xml.bind.annotation
  org.apache.commons.compress.compressors.bzip2 {version=[1.4,2)}
  org.apache.commons.compress.compressors.xz {version=[1.4,2)}

  org.apache.commons.compress.utils

  {version=[1.4,2)}

  org.apache.commons.jexl2

  {version=[2.1,3)}
• snp *

As noted in previous comments, commons-compress-1.5.jar is available already in the IGB run-time. But commons-jexl-2.1.1.jar and commons-logging-1.2.jar are not.

I have a doubt:

I am not sure if the parts of htsdjk-igb that are actually being used in IGB are needing to access functionality of the commons-jexl-2.1.1.jar bundle (and its declared requirement bundle commons-logging-1.2.jar). If yes, this means that we could declare packages from commons-jexl-2.1.1.jar as optional in the MANIFEST of htsjkd-igb.

That is, we could change the import statement for org.apache.commons.jexl2 to look like:

• org.apache.commons.jexl2;resolution:=optional;version="[2.1,3)"

instead of

• org.apache.commons.jexl2;version="[2.1,3)"

In addition, we would want to no longer export this same library from htsdjk-igb bundle.

The htsdjk-igb bundle MANIFEST can be re-built by modifying build.gradle in our repository https://github.com/lorainelab/htsjdk/tree/igb-2.16.2 (a branch)

Or we could daringly edit it by hand and see what happens.

Suggestion:

• Clone the repo
• Investigate options for osgi plug-in being used to build the bundle (to avoid building by hand if too inconvenient)
• For testing, build the bundle (jar file) and check syntax of MANIFEST using command line bnd as above

I used

• bnd print -u htsjdk-igb-2.16.2.jar

to identify packages within htsjdk that are using org.apache.commons.jexl2 artifact.

The only package in htsjdk that uses org.apache.commons.jexl2 is:

• htsjdk.variant.variantcontext

Our build.gradle file is using plugin osgi.

According to:
https://github.com/bndtools/bnd/blob/master/biz.aQute.bnd.gradle/README.md#gradle-plugin-for-workspace-builds

we should biz.aQute.bnd.builder plugin instead.

The above link contains example syntax for adding stuff to the MANIFEST.

For example:

jar {
manifest

}

[~aloraine]
According to my understanding jexl2 is used by htsjdk-igb (htsjdk.variant.variantcontext). Does it mean we cannot keep jexl2 as optional or am I wrong?

htsjdk.variant.variantcontext is extensively using jexl.
jexl is used in below java files.
1. src\main\java\htsjadk\variant\varaintcontext\JEXLMap.java
2. src\main\java\htsjadk\variant\varaintcontext\JexlMissingValueTreatment.java
3. src\main\java\htsjadk\variant\varaintcontext\VariantContextUtils.java
4. src\main\java\htsjadk\variant\varaintcontext\VaraintJEXLContext.java
5. src\main\java\htsjadk\variant\varaintcontext\VaraintJEXLContextUnitTest.java
6. src\main\java\htsjadk\variant\varaintcontext\GenotypeJEXLContext.java

Now the issue is htsjdk is exporting (2.16.2) and importing(2.16.3) the same library. I will look into it.

Branch: https://bitbucket.org/svallapu/charan_igb/branch/IGBF-1767#diff

Pull request submitted.

Works as expected.