Update IGB java 1.8 dependencies

Side note:

While attempting to install netbeans version 11.3, I ran into a bug where the installer will fail because it cannot find the jdk. The steps here fixed the issue.

The problem seems to be the fact that /usr/libexec/java_home seems to be broken in BigSur (see here: https://developer.apple.com/forums/thread/666681).

The Netbeans installer checks if the JDK is installed by using /usr/libexec/java_home, gets the wrong answer and quits with an error.

One way to fix that behaviour is to patch the installer. It can be done as follows:

assuming there is no directory called "a" in tmp:

Open the dmg, but do not run the installer

In a shell give the following command:

pkgutil --expand /Volumes/NetBeans\ 8.2\ RC/NetBeans\ 8.2\ RC.pkg /tmp/a

(if your Netbeans installer has a different name, change the command to reflect the actual name of your installer)

This opens the package, and copies it s content in /tmp/a.

Edit the installation script. Using a text editor, open the file /tmp/a/Distribution.

on line 13, replace:

var java_result = checkJavaVersion();

with

var java_result = true;

and save the file. This will skip the check on the Java version. It is your responsibility to make sure you have JDK 8 or following on your machine, and that this version is used when you use the java command. You can make sure that's fine by checking with the command

java -version

Let's wrap it up:
pkgutil --flatten /tmp/a ~/a.pkg

At this point you have in your home directory a package called a.pkg. Double click on it, and the Netbeans installer will do its job.

Seeing this error when updating org.apache.felix.scr

DEBUG: Candidate permutation failed due to a conflict between an export and import; will try another if possible. (Uses constraint violation. Unable to resolve resource org.apache.felix.http.bundle [org.apache.felix.http.bundle [62](R 62.0)] because it exports package 'org.eclipse.jetty.io' and is also exposed to it from resource org.apache.felix.http.jetty [org.apache.felix.http.jetty [63](R 63.0)] via the following dependency chain:

  org.apache.felix.http.bundle [org.apache.felix.http.bundle [62](R 62.0)]
    import: (&(osgi.wiring.package=org.eclipse.jetty.http)(version>=9.2.0)(!(version>=10.0.0)))
     |
    export: osgi.wiring.package=org.eclipse.jetty.http; uses:=org.eclipse.jetty.io
  org.apache.felix.http.jetty [org.apache.felix.http.jetty [63](R 63.0)]
    import: (&(osgi.wiring.package=org.eclipse.jetty.io)(version>=9.4.0)(!(version>=10.0.0)))
     |
    export: osgi.wiring.package: org.eclipse.jetty.io
  org.apache.felix.http.jetty [org.apache.felix.http.jetty [63](R 63.0)])

Branch: https://bitbucket.org/nfreese/nowlanfork-igb/branch/IGBF-3335

Commit c30345f is the last commit where IGB will still start and all bundles are resolved.

The upgrade of the OSGI annotations from the deprecated bnd annotations seems to be a primary issue with IGB not starting. I can resolve all bundles as visible in the felix web console, but with the new annotations IGB does not start.

Not sure if this helps or not:

One possibility may be that the new annotations are not gettin properly translated into the correct XML configuration files within the bundles.

Using commit c30345f I can start IGB through the terminal (java -jar igb_exe.jar) and I have 95 bundles total - all 95 bundles active in the felix web console.

Using commit c30345f I can start IGB through Netbeans (which uses development mode, see IGBF-1366) - 102 bundles total, 100 bundles active, 1 active fragments, 1 bundles installed.

• Apache Felix Dependency Manager Shell -> installed
• Core - Logging Config (org.lorainelab.igb.logging-config) -> fragment

For comparison, IGB release run through installer has 94 bundles total, 94 bundles active.
IGB release run through Netbeans (development mode) has 101 bundles, 101 bundles active.

Investigating why development mode is causing issues for branch IGBF-3335 commit c30345f.

The Apache Felix Dependency Manager Shell cannot resolve the imported package:
org.apache.felix.dm.diagnostics,version=[1.1,2) – Cannot be resolved

If I downgrade felixDependencyManagerShellVersion back to 3.2.0 (what it was in IGB release 9.1.10) I now see: org.apache.felix.service.command,version=[0.10,1) – Cannot be resolved

If I update org.apache.felix.dependencymanager to 4.6.0 and add a dependency on org.osgi.service.metatype then everything resolves (development mode or normal) except the following:

• org.lorainelab.igb.remote.services.discovery.provider.configured
• org.lorainelab.igb.remote.services.distribution.provider.http
• org.lorainelab.igb.remote.services.topology.manager

All three dependencies declare that they cannot use a version of org.apache.felix.dm greater than 4.0.0. I am looking for additional documentation about how to update these dependencies. There is some info in IGBF-1213 but I have not been able to find the files in the google drive.

If I create a new bundle for org.lorainelab.igb.remote.services.discovery.provider.configured by manipulating the manifest and pom to version 1.1.0 where org.apache.felix.dm can be version >=4 then the bundle is resolved, but I see:

Activator start error in bundle org.lorainelab.igb.remote.services.discovery.provider.configured
java.lang.NoSuchMethodError: org.apache.felix.dm.Component.add

This may be due to the declarative services annotations needing to be updated (I am using the older original annotations on this branch).

Settings for adding updated jar to local m2 repository:

mvn install:install-file \
   -Dfile=/Users/lorainelab/Downloads/org.lorainelab.igb.remote.services.discovery.provider.configured-1.1.0.jar \
   -DgroupId=org.lorainelab.igb \
   -DartifactId=org.lorainelab.igb.remote.services.discovery.provider.configured \
   -Dversion=1.1.0 \
   -Dpackaging=jar \
   -DgeneratePom=true

Downgrading several of the felix dependencies fixed the unresolved bundles issue.

<felixRuntimeVersion>0.16.2</felixRuntimeVersion>
<felixGogoShellVersion>0.12.0</felixGogoShellVersion>
<felixGogoCommandVersion>0.16.0</felixGogoCommandVersion>

Next steps are to squash commits, revert changes to jide (they are not related to this ticket), and to write steps for testing.

Branch: https://bitbucket.org/nfreese/nowlanfork-igb/branch/IGBF-3335

Note that we may be able to upgrade these versions along with the felix dependency manager/shell if we are able to modify/remove dependencies mentioned in IGBF-3382.

A few comments:

• this branch seems like it ought to become the new main line of development for IGB running under JDK 1.8.
• instead of merging it into branch "master", let's add the branch to the team repository
• let's name the branch something like: "main-jdk1.8" or "main-JDK8" (something welcoming to new developers)
• let's also halt development on "master"

Branch: https://bitbucket.org/nfreese/nowlanfork-igb/branch/IGBF-3335

To test:

1. Download the IGBF-3335 installer: https://bitbucket.org/nfreese/nowlanfork-igb/downloads/
2. Install and start IGB
3. Check the IGB log for errors
4. Launch the App Manager (Plug-ins tab) in IGB, check that you can install an app
5. Open a genome and load some data, i.e. check that IGB appears to be functioning correctly in a general sense
6. In a web browser, go to http://localhost:7080/system/console/bundles and check to see that all bundles are active (should be 95 bundles, check that no bundles are listed as installed, fragments)

To test development version:

1. On your fork on your local machine, fetch the IGBF-3335 branch

   git fetch https://bitbucket.org/nfreese/nowlanfork-igb.git IGBF-3335:IGBF-3335

2. Run mvn clean install to build igb
3. In an IDE, run IGB (this should run IGB in development mode, if not message Nowlan)
4. Check the IGB log for errors
5. Launch the App Manager (Plug-ins tab) in IGB, check that you can install an app
6. Open a genome and load some data, i.e. check that IGB appears to be functioning correctly in a general sense
7. In a web browser, go to http://localhost:7080/system/console/bundles and check to see that all bundles are active (should be 102 bundles, check that no bundles are listed as installed, fragments)

I reset my preferences to defaults and deleted my ".igb" bundles directory to ensure a completely fresh start for the IGBF-3335 installer, downloaded from the above link: https://bitbucket.org/nfreese/nowlanfork-igb/downloads/.

I installed the IGBF-3335 branch and ran it. I observed no errors in the log tab during startup. I clicked the Mona Lisa painting shortcut image to load the latest human genome release into IGB. I also observed zero errors in the log.

Lastly, I visited the link http://localhost:7080/system/console/bundles. This link required me to enter a user name and password. I entered "admin" as the user name and "admin" as the password. Next, the Web page titled "Apache Felix Web Console - Bundles" loaded into my browser. It listed 94 bundles, all active.

A comment: I had forgotten how useful this Web Console is! I just now realized that I can select a bundle and view information about its Manifest, its exported packages, and its imported packages, along with their accepted versions. This is very cool

I next attempted to install an IGB App, as follows:

• Selected Tools > Open App Manager
• Observed that several Apps were available, but not ProtAnnot
• Selected the top App listed in the display: "Geometric Mean Operator"
• Observed "java.io.IOException" was triggered in the IGB log tab when I selected the App.

Here is the error observed:

Jul 12, 2023 12:19:23 PM com.sun.webkit.network.URLLoader doRun
WARNING: Unexpected error
java.io.IOException: Resource does not exist: bundle://38bfba1d-676b-4ae6-8419-86d993ba80e1_77.0:0/pluginInfoTemplateResources/fonts/fontawesome-webfont.woff?v=4.4.0
at org.apache.felix.framework.URLHandlersBundleURLConnection.<init>(URLHandlersBundleURLConnection.java:135)
at org.apache.felix.framework.URLHandlersBundleStreamHandler.openConnection(URLHandlersBundleStreamHandler.java:66)
at java.net.URL.openConnection(URL.java:1027)
at com.sun.webkit.network.URLLoader.doRun(URLLoader.java:162)
at com.sun.webkit.network.URLLoader.lambda$run$0(URLLoader.java:132)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.webkit.network.URLLoader.run(URLLoader.java:131)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

Jul 12, 2023 12:19:23 PM com.sun.webkit.network.URLLoader doRun
WARNING: Unexpected error
java.io.IOException: Resource does not exist: bundle://38bfba1d-676b-4ae6-8419-86d993ba80e1_77.0:0/pluginInfoTemplateResources/fonts/fontawesome-webfont.ttf?v=4.4.0
at org.apache.felix.framework.URLHandlersBundleURLConnection.<init>(URLHandlersBundleURLConnection.java:135)
at org.apache.felix.framework.URLHandlersBundleStreamHandler.openConnection(URLHandlersBundleStreamHandler.java:66)
at java.net.URL.openConnection(URL.java:1027)
at com.sun.webkit.network.URLLoader.doRun(URLLoader.java:162)
at com.sun.webkit.network.URLLoader.lambda$run$0(URLLoader.java:132)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.webkit.network.URLLoader.run(URLLoader.java:131)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

Jul 12, 2023 12:19:23 PM com.sun.webkit.network.URLLoader doRun
WARNING: Unexpected error
java.io.IOException: Resource does not exist: bundle://38bfba1d-676b-4ae6-8419-86d993ba80e1_77.0:0/pluginInfoTemplateResources/fonts/fontawesome-webfont.svg?v=4.4.0#fontawesomeregular
at org.apache.felix.framework.URLHandlersBundleURLConnection.<init>(URLHandlersBundleURLConnection.java:135)
at org.apache.felix.framework.URLHandlersBundleStreamHandler.openConnection(URLHandlersBundleStreamHandler.java:66)
at java.net.URL.openConnection(URL.java:1027)
at com.sun.webkit.network.URLLoader.doRun(URLLoader.java:162)
at com.sun.webkit.network.URLLoader.lambda$run$0(URLLoader.java:132)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.webkit.network.URLLoader.run(URLLoader.java:131)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

However, I was able to click "Install" and install the App, as recorded in the log following my click:

12:23:22.667 INFO c.a.igb.util.IGBTrustManager - Trusted Certificates:CN=*.bioviz.org; CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB; CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
12:23:23.703 INFO c.a.igb.util.IGBTrustManager - Trusted Certificates:CN=*.s3.amazonaws.com; CN=Amazon RSA 2048 M01,O=Amazon,C=US; CN=Amazon Root CA 1,O=Amazon,C=US; CN=Starfield Services Root Certificate Authority - G2,O=Starfield Technologies\, Inc.,L=Scottsdale,ST=Arizona,C=US
12:23:24.046 INFO o.l.i.p.manager.BundleActionManager - Installed app: geometric-mean,9.1.0
12:23:24.059 INFO com.affymetrix.igb.IGB - bundle geometric-mean;9.1.0 installed

Also, when I clicked on other Apps in the App Manager display, the previously observed error did not recur.

Issue appears to be resolved by downgrading org.apache.felix.main from 6.0.4 to 6.0.3. Seems odd that a patch update leads to this error, maybe it was a bug? Will do additional testing to see if any other issues arise.

Working on testing "development" mode.

• Obtained test branch. Build it on the command line with "mvn clean install"
• Tried to run igb using the simplest possible method by entering "java -jar igb_exe.jar"
• Observed that IGB launched, not using Apple "look and feel" conventions (I'm on a Mac) because I did not tell it to do so, which I could have done by including option "-Dapple.laf.useScreenMenuBar=true" (see "run_igb.sh")
• IGB seemed to run perfectly well. The final line of the Log was: "14:22:14.145 [main] INFO com.affymetrix.main.OSGiHandler - OSGi is started with org.apache.felix.framework version 5.2.0"

Question for Nowlan Freese:

• Do you know how to run IGB in "development mode" from the command line, e.g., is there a "development mode" option?

Found it!

In main/nb_actions.xml.

<properties>
<exec.executable>java</exec.executable>
<exec.mainClass>com.affymetrix.main.Main</exec.mainClass>
<exec.args>-Xmx1g -DdevelopmentMode=true -Dapple.laf.useScreenMenuBar=true -Dsun.java2d.opengl=true -classpath %classpath com.affymetrix.main.Main</exec.args>
</properties>

Tested IGB running in Development Mode by executing:

java 
   -Xmx1g -DdevelopmentMode=true 
   -Dapple.laf.useScreenMenuBar=true 
   -Dsun.java2d.opengl=true 
   -jar igb_exe.jar >out.txt 2>err.txt

• Opened default human genome.
• Installed an app using App Manager (internal to IGB).
• Visited App Store page for the App installed in the previous step. Observed that page showed the App as currently installed.
• Visited page for ProtAnnot. The page content was nonsensical. It told me to start IGB to install and use the App, but IGB is already running. Will report this error in another issue as this is likely to not be related to the current changes. To confirm, I can check to see if the same error occurs using IGB from the not-yet-merged master branch.
• Checked felix Web console. All bundles' status values were listed as "active" except "Core - Logging Config org.lorainelab.igb.logging-config" which was instead listed as status "fragment"

Checked contents of out.txt and err.txt:

• err.txt was empty
• out.txt showed no errors or warnings

Nowlan Freese: I feel this is ready for a PR. If you have any questions or concerns let me know.

I downgraded org.apache.felix.main from 6.0.4 to 6.0.3 which appears to resolve the java.io.IOException: Resource does not exist error. I have squashed the change into the Update project pom dependencies commit and pushed the change to my branch.

It is interesting that "Core - Logging Config org.lorainelab.igb.logging-config" was listed as fragment. I have seen this occur before when I was trying to get all bundles to active by downgrading dependencies, but I have not seen it on my system since the recent changes.

I'm comfortable with moving forward with a pull request.

[~aloraine] - how would you like to handle the pull request? I think if you make a new branch "main-JDK8" then I can set it as the destination branch for the pull request.

OK - I will make the target branch and push it to the team repository shortly.

I have pushed the new development branch. It is ready to receive your PR.

See:

https://bitbucket.org/lorainelab/integrated-genome-browser/src/main-JDK8/

Pull request: https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/934

Merged and the new main-JDK8 branch installers are deployed to BioViz early access section.

Tested in local. all 95/95 bundles are active, I am able to load data and everything else looks fine. closing this ticket