Details
-
Type:
Task
-
Status: Closed (View Workflow)
-
Priority:
Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: 9.1.10 Major Release
-
Labels:
-
Story Points:4
-
Epic Link:
-
Sprint:Spring 9 2022 May 9, Summer 3 2022 June 21, Summer 4 2022 July 4, Summer 5 2022 July 18, Summer 6 2022 Aug 1, Fall 1 2022 Aug 15
Description
The newest version of Install4J allows packaging JREs that will work on both types of Apple processor / OS. Since our current release of IGB cannot function on Apple Silicon (M1 chip), as discussed in IGBF-3104, we need to provide a new installer for users with this new type of computer.
For this task:
- Understand the new version 9 Install4J features relevant to our goals
- Recreate the Docker image we use to build IGB installers via Bitbucket pipelines
- Reconfigure our bitbucket pipeline process to use the new Docker image
- Create new installers for each platform, including the new MacOS platform
- Test the new installer on a new M1 Apple computer
References:
- Docker for IGB git repository - https://bitbucket.org/lorainelab/integrated-genome-browser-docker
- Testing the Docker image interactively - https://canvas.instructure.com/courses/1164217/pages/learn-to-use-igb-docker-image-build-environment
- Cleaning up old un-needed images from a host - https://www.baeldung.com/ops/docker-removing-images
Attachments
Issue Links
- relates to
-
IGBF-3104 Investigate IGB crash on MacOS
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- To-Do
-
-
IGBF-3140 Fix App Manager problem with Zulu Java8 JDK
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
Activity
Two JRE's for Java 8 available via the Install4J have Apple Silicon support:
- Zulu - 8u332 w/ and w/o JavaFx includes: linux-amd64, linux-x86, macos-aarch64, macos-amd64, windows-amd64, windows-x86
- Liberica - 8u332+9 includes linux-aarch64, linux-aarch64-musl, linux-amd64, lunux-amd64-musl, linux-ppc64le, linux-x86, macos-aarch64, macos-amd64, solaris-amd64, solaris-sparcv9, src, windows-amd64, windows-x86
macos-amd64 - Intel chip
macos-aarch64 - Apple Silicon chip
Trying Zulu JRE, downloaded from:
https://www.azul.com/downloads/?version=java-8-lts&os=macos&architecture=x86-64-bit&package=jre-fx
Built installers using Zulu JRE. However, the notarization step failed for some reason.
CLI:
xcrun altool --notarize-app --primary-bundle-id "2022-05-16-igb-9.1.10" --username XXX --password XXX --asc-provider XXX --file IGB-macos-9.1.10.dmg
ninfo=XXX # value returned by previous command
xcrun altool --notarization-info $ninfo -u XXX -p XXX
xcrun stapler staple IGB-macos-9.1.8.dmg
Tried again a few days later. It worked a bit more this time. Used new app password and apparently the upload was accepted at apple. Result:
Using XCode tools from /Applications/Xcode.app/Contents/Developer Uploading IGB-macos-9.1.10.dmg for notarization Waiting for notarization result Notarized /Users/aloraine/src/integrated-genome-browser/distribution/IGB-macos-9.1.10.dmg
Made new 9.1.8 release installer using Zulu JRE, notarized. Tested briefly on my Intel MacOS. Application started without a problem. Now need to test it on an Apple Silicon Mac.
Uploaded the installer as "Zulu-IGB-macos-9.1.8.dmg" to Downloads section of my fork.
To test this on an Apple Silicon computer, download from:
https://bitbucket.org/aloraine/integrated-genome-browser/downloads/Zulu-IGB-macos-9.1.8.dmg
attn: Nowlan Freese
Initial testing on the new M1 Macbook Air.
The M1 came with Monterey 12.2.1 installed. Users have reported issues with IGB on Monterey 12.3.1. Using IGB release 9.1.8 I have not encountered any issues with IGB locking up.
I have tried to install the Zulu-IGB-macos-9.1.8.dmg but am running into an issue where the Mac states that "You need at least a 1.8 Java Virtual Machine to install Integrated Genome Browser". This can be bypassed with the following steps:
- Press CMD + I on the Integrated Genome Browser Installer icon. This should bring up an options panel.
- In the options panel there should be a checkbox with "Open using Rosetta". Check the box and then try running the installer.
- Once IGB is installed you will need to check the box to "Open using Rosetta" for the IGB icon.
However, this solution seems confusing for users.
Next I am going to upgrade from 12.2.1 to 12.4 to see if the issues persist.
User has reported that using the new installer (zulu) IGB was still crashing on their system.
Thank you Nowlan Freese for the detailed reports in above comments.
The installer I made and which you tested in the comments above did not include a Java Virtual Machine (JRE) due to a mistake in how I configured the Install4J project.
I have corrected this and re-uploaded the installer to the same link as above: https://bitbucket.org/aloraine/integrated-genome-browser/downloads/Zulu-IGB-macos-9.1.8.dmg
For the next steps:
- Please un-install IGB from the new M1 Mac
- Download the new installer and use it to install IGB onto the M1 Mac
No errors should result.
The updated installer worked correctly as I did not encounter the popup regarding the Java Virtual Machine. I did not encounter any crashes in IGB on Mac OS 12.2.1.
My next step will be to update to Mac OS 12.4 and see if IGB has any issues.
On Mac OS 12.4 I had no issues running IGB 9.1.8 release and the Zulu-IGB. Neither version crashed after interacting with them for around 10 minutes, including loading data and using track operations.
I am going to reach out to the user that was having issues and see if I can further diagnose this issue.
As the MacOS problems look like they may be resolved, am now working on creating a new Docker image to build IGB installers using Bitbucket.
I reached out to the user with instructions for resetting preferences to default, removing the .igb directory, and uninstalling igb prior to installing the latest Zulu-IGB. However, the user has responded that they may not have time over the next week.
After discussing with Dr. Loraine, the issue this user is encountering may be related to an incompatible bundle cache, though it is unclear how this would happen. As uninstalling IGB does not remove the bundle cache (IGBF-3120) removing the .igb/bundles directory may be required to correct this issue.
I will spend some additional time on the M1 Mac attempting to replicate the issue.
Considering deploying the new MacOS Zulu-java installer to BioViz for new users to get. And if old users have trouble with their older version, they might try un-intalling and re-installing.
An additional user has encountered issues with IGB:
I recently upgraded my MacOS from El Capitan (10.11) to Monterey (12) and IGB stopped working. I have been using IGB for many years before that without major issues. Attached is a text file with the system info, and the error log. In most cases, IGB already crashes before opening (‘quit unexpectedly”), sometimes it opens ok at first, but thencrashes as soon as I try to load data files. Any help would be appreciated. (I submitted the same crash report using the feedback form, but am not sure whether it got through as I don’t have a JIRA account.)
Update as of June 21, 2022:
- Created new JRE/JDK bundles for 64-bit windows, Linux, Apple OS
- Decided to no longer support 32-bit windows
- Planning to use Zulu JDK/JRE builds for all three platforms, comparative testing in progress (see linked tickets)
Note: Installers for testing are available with "Zulu" file name prefix from https://bitbucket.org/lorainelab/integrated-genome-browser/downloads/
- So far, Zulu MacOS installers with so-called universal binaries are working fine for all users who contacted us
For next steps, in addition to updating the Docker image, we need to do the following
- Update docker build file, run it, check it works, increment version
- Release new versions of 9.1.8 for MacOS, requires updating ansible playbooks for bioviz to accommodate this special case release
- Create new version of bitbucket build file (challenging to get it right)
- Carefully update install4jv9 configuration file to ensure the right IGB version included (can this be automated?)
For some reason, IGB has some code that is trying to access updates.xml for some reason. Why? That file is meant to enable Install4J to update the application independently of IGB code - I thought. Look into this.
Our plugin is: "update"
Investigate different JVM to deal with possible javascript problem observed during testing Zulu installers / IGB.
Install4J has an API that allows you to interact with the launcher, programmatically. Here is some documentation about that:
https://www.ej-technologies.com/resources/install4j/v/9.0/help/doc/api/launcher.html?light
Now trying the Liberica JVM build with Install4J:
Collecting files:
replaced variable 'projectHomeDirectory' with '..' in string '${compiler:projectHomeDirectory}'
Copying /Users/aloraine/src/aloraine-integrated-genome-browser/distribution/..(existing: true, excluded: false) to
/Users/aloraine/src/aloraine-integrated-genome-browser/CodeOfConduct.md
/Users/aloraine/src/aloraine-integrated-genome-browser/LICENSE.html
/Users/aloraine/src/aloraine-integrated-genome-browser/NOTICES.txt
/Users/aloraine/src/aloraine-integrated-genome-browser/README.md
/Users/aloraine/src/aloraine-integrated-genome-browser/igb_exe.jar
Creating JRE bundle:
Finding JDK release 8/8u332+9 [macos-universal]
Finding JDK release 8/8u332+9 [macos-aarch64]
Downloading bellsoft-jdk8u332%2B9-macos-aarch64-full.zip (100%)
Processing downloaded JDK archive
Extracting JRE
Configuring Pack200
Finding JDK release 8/8u332+9 [macos-amd64]
Downloading bellsoft-jdk8u332%2B9-macos-amd64-full.zip (100%)
Processing downloaded JDK archive
Extracting JRE
Packing lib/rt.jar
Packing lib/jfr.jar
Packing lib/jsse.jar
Packing lib/jfxswt.jar
Packing lib/charsets.jar
Packing lib/resources.jar
Packing lib/management-agent.jar
Packing lib/jce.jar
Packing lib/ext/sunec.jar
Packing lib/ext/nashorn.jar
Packing lib/ext/cldrdata.jar
Packing lib/ext/jfxrt.jar
Packing lib/ext/dnsns.jar
Packing lib/ext/localedata.jar
Packing lib/ext/sunjce_provider.jar
Packing lib/ext/sunpkcs11.jar
Packing lib/ext/jaccess.jar
Packing lib/ext/zipfs.jar
Writing JRE bundle archive (100%)
Configuring Pack200
Packing lib/rt.jar
After building the installer, I checked App Manager. The App Manager window contained an empty panel.
Thanks to work done on IGBF-3140, we can now distribute IGB for MacOS with the "zulu" JavaFX-containgin, universal binary JRE bundled with it.
So, I am now working on created a new Docker image we can use for this in conjunction with bitbucket pipelines.
To start, I cloned https://bitbucket.org/lorainelab/integrated-genome-browser-docker onto my local computer so that I could run the ansible playbooks for updating our docker host VM in AWS.
I ran the playbook with:
inventory.ini
ansible-playbook -i inventory.ini install.yml
using inventory.ini:
[dockerhosts]
example1 ansible_host=54.237.97.202 ansible_ssh_private_key_file=~/.ssh/pems/dockerhost.pem
[dockerhosts:vars]
ansible_ssh_common_args="-o StrictHostKeyChecking=no"
ansible_python_interpreter=/usr/bin/python
ansible_ssh_user=ec2-user
In addition, I added my new computer's public key to the "pubkey" role, which means I won't need to include "ansible_ssh_private_key_file" option in future playbook runs.
Now following instructions written by Noor Zahara in this comment:
Steps followed to create docker image :
1. Download and install docker software
2. Once Docker file from IGB docker repository has been modified as per the requirement, go to that directory.
3. Build the docker image from docker file using docker build -t <image_name> <path>.
4. run docker ps -a to get the container id.
5. Login to docker hub using the command Docker login -u <username> -p <password>
6. Commit the changes using Docker commit <container-id> <imagename:tag>
7. Finally push the changes - Docker push <imagename:tag>Note - Testers shouldn't have to push anything to docker hub. The above instructions are for creating a docker image.
Once the installer is installed, 'release' file inside the 'jre' folder can be checked to verify the version.
Created new public key for the Dockerhost VM. Added this to my bitbucket account. Forked integrated-genome-browser-docker into my account. Cloned the fork to Dockerhost with command
git clone git@bitbucket.org:aloraine/integrated-genome-browser-docker.git
Made a new branch for this ticket on the clone.
Edited the Dockerfile.
image name: igb-maven-install4j
image tag: I4J9-jre-1.8.0_332
Running command:
docker build -t igb-maven-install4j:I4J9-jre-1.8.0_332 .
Result:
Successfully built 2bbcf9db93a0
Successfully tagged igb-maven-install4j:I4J9-jre-1.8.0_332
Checked image with:
dockerhost ec2-user $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE igb-maven-install4j I4J9-jre-1.8.0_332 2bbcf9db93a0 2 minutes ago 1.89GB apache latest 86f6725602ce 17 months ago 333MB <none> <none> d3674b9a56d6 17 months ago 333MB composetest_web latest bee1bcb4e669 17 months ago 182MB redis alpine 4fb69be5871b 17 months ago 32.3MB mysql latest 8457e9155715 17 months ago 546MB python 3.7-alpine 79d8d54b1857 17 months ago 41.6MB ubuntu latest f63181f19b2f 18 months ago 72.9MB centos latest 300e315adb2f 19 months ago 209MB lorainelab/igb-maven-install4j latest af17d5a8f332 21 months ago 1.88GB hello-world latest bf756fb1ae65 2 years ago 13.3kB maven 3.5.0 080cf1fccf1c 4 years ago 750MB tecadmin/ubuntu-ssh 16.04 bb63b492da01 5 years ago 223MB
Checking the image with:
docker run -i -t -v ~/build:/build --workdir /build igb-maven-install4j:I4J9-jre-1.8.0_332
Confirmed that the image contains Azul's Zulu JVM in /docker-java-home.
Executing java commands:
root@183dce3a9dcd:/docker-java-home# java -version
openjdk version "1.8.0_342"
OpenJDK Runtime Environment (Zulu 8.64.0.15-CA-linux64) (build 1.8.0_342-b07)
OpenJDK 64-Bit Server VM (Zulu 8.64.0.15-CA-linux64) (build 25.342-b07, mixed mode)
The only commands that are in the user's path are ones where the Docker build command included the "update-alternatives" line from the Dockerfile. This means that not all java commands available in "bin" will be available in the container, for example, keytool won't be available. Is this going to be a problem? This seems a little cheesy. What's weird, however, is that jarsigner I believe is run by our build process. Why did this not fail before? Was it because our installation of install4J has its own java? Checking this...
Here is where install4j ended up:
root@183dce3a9dcd:/usr/local/bin# ls -lh total 8.0K lrwxrwxrwx 1 root staff 32 Jul 29 22:21 createbundle -> /opt/install4j9/bin/createbundle -rwxr-xr-x 1 root staff 87 Oct 10 2017 docker-java-home lrwxrwxrwx 1 root staff 29 Jul 29 22:21 install4j -> /opt/install4j9/bin/install4j lrwxrwxrwx 1 root staff 30 Jul 29 22:21 install4jc -> /opt/install4j9/bin/install4jc -rwxrwxr-x 1 root root 1.1K Oct 10 2017 mvn-entrypoint.sh lrwxrwxrwx 1 root staff 27 Jul 29 22:21 updater -> /opt/install4j9/bin/updater
Here's maven info:
root@183dce3a9dcd:~/.m2# mvn -v Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T19:39:06Z) Maven home: /usr/share/maven Java version: 1.8.0_342, vendor: Azul Systems, Inc. Java home: /usr/lib/jvm/zulu8.64.0.15-ca-fx-jdk8.0.342-linux_x64/jre Default locale: en, platform encoding: UTF-8 OS name: "linux", version: "4.14.287-215.504.amzn2.x86_64", arch: "amd64", family: "unix"
Now to test that the container can build IGB, following the instructions in "Learn to use IGB Docker image build environment" !
Not yet. Looks like a tarball got incorrectly copied into the donctainer:
root@183dce3a9dcd:~# ls .install4j9/jres/ linux-amd64-1.8.0_332.tar.gz macos-universal-1.8.0_332.tar.gz windows-amd64-1.8.0_332.tar.gz zulu8.64.0.15-ca-fx-jdk8.0.342-linux_x64.tar.gz
Rebuilt the image after editing the Dockerfile. Looks like the old image is still there but lost its name and tag, probably because the new one now has its name and tag:
dockerhost ec2-user $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE igb-maven-install4j I4J9-jre-1.8.0_332 c81ffbbdee01 2 days ago 1.73GB <none> <none> 2bbcf9db93a0 2 days ago 1.89GB apache latest 86f6725602ce 17 months ago 333MB <none> <none> d3674b9a56d6 17 months ago 333MB composetest_web latest bee1bcb4e669 17 months ago 182MB redis alpine 4fb69be5871b 17 months ago 32.3MB mysql latest 8457e9155715 17 months ago 546MB python 3.7-alpine 79d8d54b1857 17 months ago 41.6MB ubuntu latest f63181f19b2f 18 months ago 72.9MB centos latest 300e315adb2f 20 months ago 209MB lorainelab/igb-maven-install4j latest af17d5a8f332 21 months ago 1.88GB hello-world latest bf756fb1ae65 2 years ago 13.3kB maven 3.5.0 080cf1fccf1c 4 years ago 750MB tecadmin/ubuntu-ssh 16.04 bb63b492da01 5 years ago 223MB
Used a post from Baeldung to learn about removing unwanted Docker images.
Removed unused images and stopped containers:
dockerhost ec2-user $ docker system df --format 'table {{.Type}}\t{{.TotalCount}}\t{{.Size}}'
TYPE TOTAL SIZE
Images 6 2.095GB
Containers 0 0B
Local Volumes 7 426.1MB
Build Cache 0 0B
Checking the rebuild container (from previous comment):
dockerhost ec2-user $ docker run -i -t -v ~/build:/build --workdir /build igb-maven-install4j:I4J9-jre-1.8.0_332 root@4b6b763a6875:/build#
Note: Image is 1.73 Gb.
Inside root (home) directory for bash default user in the image (root?):
root@4b6b763a6875:~# find . . ./.bashrc ./.profile ./.install4j9 ./.install4j9/jres ./.install4j9/jres/linux-amd64-1.8.0_332.tar.gz ./.install4j9/jres/macos-universal-1.8.0_332.tar.gz ./.install4j9/jres/windows-amd64-1.8.0_332.tar.gz ./.wget-hsts ./.m2 ./.m2/settings.xml ./.m2/copy_reference_file.log ./.m2/settings-docker.xml
Checking to see if all this is OK.
Checking settings.xml:
root@4b6b763a6875:~# more .m2/settings.xml <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd"> <servers> <!-- This file will get saved in the root user's .m2 directory in the Docker image used to build IGB. Later, when building IGB either in Bitbucket pipelines or somewhere else, defining these values for the maven environment will allow the user to run the "deploy" profile defined in IGB's pom.xml file and deploy IGB artifacts to the repository defined below --> <server> <id>${repo.id}</id> <username>${repo.username}</username> <password>${repo.password}</password> </server> </servers> </settings>
Note sure why this is here:
root@4b6b763a6875:~# more .wget-hsts
# HSTS 1.0 Known Hosts database for GNU Wget.
# Edit at your own risk.
# <hostname> <port> <incl. subdomains> <created> <max-age>
bitbucket.org 0 1 1659135221 31536000
I guess I'm OK with it.
Checking the version of Java packaged with Install4J:
root@4b6b763a6875:/opt/install4j9/jre/bin# ls java jrunscript keytool pack200 rmid rmiregistry unpack200 root@4b6b763a6875:/opt/install4j9/jre/bin# ./java -version openjdk version "11.0.13" 2021-10-19 OpenJDK Runtime Environment JBR-11.0.13.8-1890.3-jcef (build 11.0.13+8-b1890.3) OpenJDK 64-Bit Server VM JBR-11.0.13.8-1890.3-jcef (build 11.0.13+8-b1890.3, mixed mode, sharing) root@4b6b763a6875:/opt/install4j9/jre/bin# pwd /opt/install4j9/jre/bin root@4b6b763a6875:/opt/install4j9/jre/bin# ./java -version openjdk version "11.0.13" 2021-10-19 OpenJDK Runtime Environment JBR-11.0.13.8-1890.3-jcef (build 11.0.13+8-b1890.3) OpenJDK 64-Bit Server VM JBR-11.0.13.8-1890.3-jcef (build 11.0.13+8-b1890.3, mixed mode, sharing)
I guess it's none of my business with JRE Install4J wants to use.
Testing the build process. While "inside" the container, I changed into the /build directory and typed "mvn install". (First checked in another login shell that ~build contained IGB clode.)
Running this build command triggered downloading of IGB build dependencies from maven central. The mvn command reported success and IGB's "igb_exe.jar" file appeared in the "distribution" directory. (Expected behavior.)
Next, testing the next step: packaging the jar file into an Install4J version 9 installer, together with the JREs. Note that we are going to probably need to edit IGB's Install4J configuration file and possibly also the pom.xml profile that governs how the install4J packaging happens.
To run it, you make a file containing environment variables for the container to use, as follows:
dockerhost ec2-user $ docker run --workdir /build --env-file env_vars -i -t -v ~/build:/build igb-maven-install4j:I4J9-jre-1.8.0_332
root@6b3bc62c541d:/build# printenv
LANG=C.UTF-8
HOSTNAME=6b3bc62c541d
MAVEN_HOME=/usr/share/maven
MAC_KEYSTORE_PASSWORD=Ll-2013!
JAVA_HOME=/docker-java-home
INSTALL4J_LICENSE=S-M9-INTEGRATED_GENOME_BROWSER#54582010001-7f7aag367qrr#b8bb23
JAVA_VERSION=8u141
PWD=/build
HOME=/root
CA_CERTIFICATES_JAVA_VERSION=20170531+nmu1
JAVA_DEBIAN_VERSION=8u141-b15-1~deb9u1
TERM=xterm
SHLVL=1
WIN_KEYSTORE_PASSWORD=UniversityOfMumbai
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
BASH_FUNC_copy_reference_file%%=() { local root="${1}";
local f="${2%/}";
local logfile="${3}";
local rel="${f/${root}/}";
echo "$f" >> "$logfile";
echo " $f -> $rel" >> "$logfile";
if [[ ! -e ${MAVEN_CONFIG}/${rel} || $f = *.override ]]; then
echo "copy $rel to ${MAVEN_CONFIG}" >> "$logfile";
mkdir -p "${MAVEN_CONFIG}/$(dirname "${rel}")";
cp -r "${f}" "${MAVEN_CONFIG}/${rel}";
fi
}
_=/usr/bin/printenv
Everything looks OK.
Also, need to provide name of jar-signing dependencies?
Looking into it.
Maybe I should change the tag to indicate JVM build not just JVM version.
Problem is, there are many configurations, too many to pack into a "name" and "tag" string.
Not sure what to do.
Set up environment as instructed and re-started the container. Entered:
mvn install -P release-bitbucket-pipelines
Build commenced, along with re-downloading of many dependencies. Installer did not build.
Edited main/pom.xml bitbucket pipelines profile definition to use install4jv9.install4j. Re-ran above, still no installers. Edited install4j version in the pom.xml in release-bitbucket-pipelines profile configuration. I guess the plugin failed silently previously but did attempt to run maybe?
New error message indicates the plugin tried to do something:
[INFO] IGB Main (run me) .................................. FAILURE [ 16.584 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:19 min [INFO] Finished at: 2022-08-01T20:05:43Z [INFO] Final Memory: 76M/438M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.sonatype.install4j:install4j-maven-plugin:1.1.1:compile (compile-installers) on project main: null returned: 1 -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :main
Question: need to invoke with command line parameters? Answer: Doesn't look that way, if I provide the command-line parameters as environment variables? Confused about why bitbucket pipeline file invokes mvn for profile release-bitbucket-pipelines in two different ways, one with and one without command line options.
Editing install4j version 9 config file to remove the 32-bit installer because I don't have a JRE for it.
There's a new version of the plugin. Maybe install4j version 9 needs a newer plugin. Version 1.1.2 appears to be the current one, replacing 1.1.1, itself released in or before 2015.
Reference:
Shot in the dark: replacing the plugin version!
Note: Install4j plugin version is defined in main/pom.xml and not in ./pom.xml.
Yay! An error message I understand:
[jar] Updating jar: /build/igb_exe.jar
[INFO] Executed tasks
[INFO]
[INFO] --- install4j-maven-plugin:1.1.2:compile (compile-installers) @ main ---
[INFO] install4j version 9.0.7 (build 9184), built on 2022-05-23
[INFO] Registered to
[INFO]
[INFO] install4j: compilation failed. Reason: java.io.FileNotFoundException: Could not find JRE bundle file /root/.local/share/install4j/v9/jres/windows-amd64-1.8.0_332.tar.gz
[INFO] Loading config file /build/distribution/install4jv9.install4j
[INFO] Creating media file 'Windows_x86_64':
[INFO] --------------------------------------
Changed location of JREs to /root/.local/share/install4j/v9/jres/ in Dockerfile. Rebuilt Docker image.
Recap testing process
In home directory (ec2-user), run interactive docker container session with:
docker run --workdir /build --env-file env_vars -i -t -v ~/build:/build igb-maven-install4j:I4J9-jre-1.8.0_332
Inside container, run mvn with:
mvn -DskipTests=true -P release-bitbucket-pipelines install
Installers should appear in:
- container file system: /build/main/target/media
- host file system: /home/ec2-user/build/main/target/media
Result:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:51 min
[INFO] Finished at: 2022-08-02T12:37:13Z
[INFO] Final Memory: 80M/440M
[INFO] ------------------------------------------------------------------------
media directory contents:
root@de7f8c0bea0c:/build/main/target/media# ls -lh total 445M -rw-r--r-- 1 root root 172M Aug 2 12:37 IGB-macos-9.1.10.dmg -rw-r--r-- 1 root root 141M Aug 2 12:36 IGB-unix-9.1.10.sh -rw-r--r-- 1 root root 132M Aug 2 12:36 IGB-windows-x64-9.1.10.exe -rw-r--r-- 1 root root 169 Aug 2 12:37 md5sums drwxr-xr-x 2 root root 194 Aug 1 19:57 oldstuff -rw-r--r-- 1 root root 337 Aug 2 12:37 output.txt -rw-r--r-- 1 root root 265 Aug 2 12:37 sha256sums -rw-r--r-- 1 root root 1.4K Aug 2 12:37 updates.xml
Warnings during Install4J plugin 1.1.2 execution:
[INFO] Executed tasks
[INFO]
[INFO] — install4j-maven-plugin:1.1.2:compile (compile-installers) @ main —
[INFO] install4j version 9.0.7 (build 9184), built on 2022-05-23
[INFO] Registered to
[INFO]
[INFO] Loading config file /build/distribution/install4jv9.install4j
[INFO] Creating media file 'Windows_x86_64':
[INFO] Collecting files:
[INFO] Compiling launchers:
[INFO] Compiling launcher 'IntegratedGenomeBrowser':
[INFO] Signing executable
[INFO] Generating VM options file IntegratedGenomeBrowser.vmoptions.
[INFO] Compiling launcher 'Updater with silent version check':
[INFO] Signing executable
[INFO] Creating media file:
[INFO] Using custom installer icon: /build/distribution/../distribution/branding/igb_128.ico
[INFO] Signing executable
[INFO] [WARNING] Warnings during script compilation:
[INFO] ----------
[INFO] In screen "Update message [Form]", property "Pre-activation expression":
[INFO] 1. WARNING in /tmp/script16810373880718444001.java.dir/com/install4j/script/I4jScript_Internal_26.java (at line 2)
[INFO] wizardContext.setNextButtonVisible(false);
[INFO] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] The method setNextButtonVisible(boolean) from the type WizardContext is deprecated
[INFO] ----------
[INFO] In screen "Update message [Form]", property "Pre-activation expression":
[INFO] 2. WARNING in /tmp/script16810373880718444001.java.dir/com/install4j/script/I4jScript_Internal_26.java (at line 3)
[INFO] wizardContext.setCancelButtonVisible(false);
[INFO] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] The method setCancelButtonVisible(boolean) from the type WizardContext is deprecated
[INFO] ----------
[INFO] 2 problems (2 warnings)
[INFO] Zipping custom code & resources JAR file
[INFO] Identifying components
[INFO] Adding JRE (windows-amd64-1.8.0_332.tar.gz)
[INFO]
[INFO] Creating media file 'Unix Installer':
[INFO] Collecting files:
[INFO] Compiling launchers:
[INFO] Compiling launcher 'IntegratedGenomeBrowser':
[INFO] Generating launcher script file
[INFO] Generating VM options file IntegratedGenomeBrowser.vmoptions.
[INFO] Compiling launcher 'Updater with silent version check':
[INFO] Generating launcher script file
[INFO] Creating media file:
[INFO] Generating launcher script file
[INFO] [WARNING] Warnings during script compilation:
[INFO] ----------
[INFO] In screen "Update message [Form]", property "Pre-activation expression":
[INFO] 1. WARNING in /tmp/script308209408855187650.java.dir/com/install4j/script/I4jScript_Internal_68.java (at line 2)
[INFO] wizardContext.setNextButtonVisible(false);
[INFO] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] The method setNextButtonVisible(boolean) from the type WizardContext is deprecated
[INFO] ----------
[INFO] In screen "Update message [Form]", property "Pre-activation expression":
[INFO] 2. WARNING in /tmp/script308209408855187650.java.dir/com/install4j/script/I4jScript_Internal_68.java (at line 3)
[INFO] wizardContext.setCancelButtonVisible(false);
[INFO] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] The method setCancelButtonVisible(boolean) from the type WizardContext is deprecated
[INFO] ----------
[INFO] 2 problems (2 warnings)
[INFO] Zipping custom code & resources JAR file
[INFO] Identifying components
[INFO] Adding JRE (linux-amd64-1.8.0_332.tar.gz)
[INFO]
[INFO] Creating media file 'Mac OS X Folder':
[INFO] Collecting files:
[INFO] Compiling launchers:
[INFO] Compiling launcher 'IntegratedGenomeBrowser':
[INFO] Generating VM options file vmoptions.txt.
[INFO] Signing launcher
[INFO] Compiling launcher 'Updater with silent version check':
[INFO] Signing launcher
[INFO] Creating media file:
[INFO] Signing launcher
[INFO] [WARNING] Warnings during script compilation:
[INFO] ----------
[INFO] In screen "Update message [Form]", property "Pre-activation expression":
[INFO] 1. WARNING in /tmp/script17755716253572487876.java.dir/com/install4j/script/I4jScript_Internal_110.java (at line 2)
[INFO] wizardContext.setNextButtonVisible(false);
[INFO] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] The method setNextButtonVisible(boolean) from the type WizardContext is deprecated
[INFO] ----------
[INFO] In screen "Update message [Form]", property "Pre-activation expression":
[INFO] 2. WARNING in /tmp/script17755716253572487876.java.dir/com/install4j/script/I4jScript_Internal_110.java (at line 3)
[INFO] wizardContext.setCancelButtonVisible(false);
[INFO] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] The method setCancelButtonVisible(boolean) from the type WizardContext is deprecated
[INFO] ----------
[INFO] 2 problems (2 warnings)
[INFO] Zipping custom code & resource
Copying installers to Web download location where they can be tested: https://bitbucket.org/aloraine/integrated-genome-browser/downloads/
Files are:
rw-r--r-- 1 aloraine staff 172M Aug 2 09:22 IGB-macos-9.1.10.dmg -rw-r--r-- 1 aloraine staff 140M Aug 2 09:22 IGB-unix-9.1.10.sh -rw-r--r-- 1 aloraine staff 132M Aug 2 09:22 IGB-windows-x64-9.1.10.exe
Note modification date. Install these ones for testing.
Also note: MacOS DMG (installer) is signed but not notarized.
To install on Apple - from Google:
How do I open an app from an unidentified developer?
In the Finder, locate the app you want to open. Don't use Launchpad to do this. Launchpad doesn't allow you to access the shortcut menu.
While pressing the Control key, click the app icon. Choose Open from the shortcut menu.
Click Open.
Installed Apple IGB 9.1.10. The UI is very messed up! Looks like there are two IGB instances running at once, or trying to? Not sure what the problem is. Nowlan Freese - could you try installing it as well?
Link to installer: https://bitbucket.org/aloraine/integrated-genome-browser/downloads/IGB-macos-9.1.10.dmg
Note: Screen capture of messed-up IGB is attached.
Investigating version of IGB (branch, latest commit) that the installer includes.
Testing on Mac 12.5. Reset preferences to default and deleted the .igb folder.
I am also seeing the duplicated IGB.
Log is attached.
Ran mvn clean and checked that latest version of main (master) branch is on the dockerhost.
After running mvn clean to remove all old builds, re-ran the mvn install command. Installers produced have different sizes, indicating the previously tested installers had "extra IGB" in them.
Old "extra IGB installers":
-rw-r--r-- 1 ec2-user ec2-user 172M Aug 2 13:21 IGB-macos-9.1.10.dmg -rw-r--r-- 1 ec2-user ec2-user 141M Aug 2 13:21 IGB-unix-9.1.10.sh -rw-r--r-- 1 ec2-user ec2-user 132M Aug 2 13:21 IGB-windows-x64-9.1.10.exe
Newly build IGB installers:
dockerhost ec2-user $ ls -lh total 372M -rw-r--r-- 1 root root 148M Aug 3 16:51 IGB-macos-9.1.10.dmg -rw-r--r-- 1 root root 117M Aug 3 16:50 IGB-unix-9.1.10.sh -rw-r--r-- 1 root root 108M Aug 3 16:50 IGB-windows-x64-9.1.10.exe
Moving newly build installers to my fork's download site for testing.
Double-IGB error is gone. Confirmed that the App Manager is able to install and uninstall Apps. However, there an empty file "foo" that showed up in the installation directory on my Mac. I think this is because within the top-level "build" directory, the checked-out IGB, every file gets automatically copied into the installer package, except for files that are explicitly excluded. No need to re-build the installers, I think. We can proceed with testing them.
Note for Karthik Raveendran - could you check that the Windows installer is OK? (Don't worry about the empty file "foo", but do please check that the contents of the installation directory look OK.)
cc: Nowlan Freese
Pushing the docker container to dockerhub. Started a container with "docker run" and not using the environment variable option because I don't want those variables to be available when we run docker pull because those variables need to be set up by the running environment.
Logged into docker hub using the command from Noor's comment above.
Then did:
dockerhost ec2-user $ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4d6d5e619281 igb-maven-install4j:I4J9-jre-1.8.0_332 "/usr/local/bin/mvn-…" About a minute ago Exited (0) About a minute ago stupefied_shtern
dockerhost ec2-user $ docker commit 4d6d5e619281 igb-maven-install4j:I4J9-jre-1.8.0_332
sha256:e2b20d68ca99168738e2fdc0abfbf2a6b33efc6ed90b23fad45233602fa6e4de
Tried to push, but got an error:
denied: requested access to the resource is denied
Above commands do not strictly work. Reading the documentation from Docker hub, I did this:
- Re-tagged the existing, built and committed (I think?) image with:
docker tag e2b20d68ca99 lorainelab/igb-maven-install4j:I4J9-jre-1.8.0_332
- Pushed to docker hub with:
docker push lorainelab/igb-maven-install4j:I4J9-jre-1.8.0_332
Here, "user" is "lorainelab" and "repository" is igb-maven-install4j.
Looks OK on the docker hub end - see: https://hub.docker.com/repository/docker/lorainelab/igb-maven-install4j
Documentation I used: https://docs.docker.com/docker-hub/repos/
Next steps:
- Delete the build, committed install4j images from the dockerhost
- Delect the containers also
- Pull the container from docker hub
- Check that it works
After that, move on to push the changes to my branch to my aloraine fork. Edit the bitbucket file and test it can build the branch with the new docker container.
Pushed changes to integrated-genome-browser repository to new branch on my fork. See: https://bitbucket.org/aloraine/integrated-genome-browser/branch/IGBF-3113
After checking installation folders of both 9.1.8 and 9.1.10 version of IGB, following observations were made
| 9.1.8 installation folder | 9.1.10 installation folder |
|---|---|
| 10 items | 10 items (apart from 'foo') |
| .install4j folder : 30 items | .install4j folder: 33 items ("user folder" and 2 PNG files(i4j_extf_11_plsfox_18gg8kx@2x_dark,i4j_extf_11_plsfox_18gg8kx_dark)) |
| jre folder : 9 items | jre folder : 6 items |
| jre/bin: 128 items | jre/bin: 115 items |
| jre/lib: 40 items | jre/lib : 35 items |
Testing whether I can build my branch using the revised build files (bitbucket pipelines file, etc) on my fork. But first, needed to update my "install4J" license string since the new Docker image uses version 9 of the software instead of version 8.
First attempt to build failed during a download. Looks like a "jetty" artifact was not available for some reason. Re-trying as this could be a network error of some type. I say this because the container was able to build IGB when deployed on the dockerhost EC2.
For further testing, IGB 9.1.10 was installed and run. When apps were installed in IGB the following exception appeared after the app was installed. This exception led to the whole bottom tabbed panel in IGB to freeze indefinitely.
Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
at javax.swing.plaf.basic.BasicTabbedPaneUI.scrollableTabLayoutEnabled(BasicTabbedPaneUI.java:263)
at javax.swing.plaf.basic.BasicTabbedPaneUI.tabForCoordinate(BasicTabbedPaneUI.java:1497)
at javax.swing.plaf.basic.BasicTabbedPaneUI.setRolloverTab(BasicTabbedPaneUI.java:575)
at javax.swing.plaf.basic.BasicTabbedPaneUI.access$2100(BasicTabbedPaneUI.java:54)
at javax.swing.plaf.basic.BasicTabbedPaneUI$Handler.mouseEntered(BasicTabbedPaneUI.java:3629)
at com.affymetrix.igb.window.service.def.JTabbedTrayPane$2.mouseEntered(JTabbedTrayPane.java:158)
at java.awt.AWTEventMulticaster.mouseEntered(AWTEventMulticaster.java:299)
at java.awt.AWTEventMulticaster.mouseEntered(AWTEventMulticaster.java:299)
at java.awt.Component.processMouseEvent(Component.java:6548)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
at java.awt.Component.processEvent(Component.java:6304)
at java.awt.Container.processEvent(Container.java:2239)
at java.awt.Component.dispatchEventImpl(Component.java:4889)
at java.awt.Container.dispatchEventImpl(Container.java:2297)
at java.awt.Component.dispatchEvent(Component.java:4711)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4904)
at java.awt.LightweightDispatcher.retargetMouseEnterExit(Container.java:4686)
at java.awt.LightweightDispatcher.trackMouseEnterExit(Container.java:4664)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4515)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4476)
at java.awt.Container.dispatchEventImpl(Container.java:2283)
at java.awt.Window.dispatchEventImpl(Window.java:2746)
at java.awt.Component.dispatchEvent(Component.java:4711)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:760)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:84)
at java.awt.EventQueue$4.run(EventQueue.java:733)
at java.awt.EventQueue$4.run(EventQueue.java:731)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:730)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Ann's note / comment on the above error: Let's make this a separate ticket. I don't know what's causing it, of course, but I suspect it has to do with the code and not the build environment or installer. attn: Nowlan Freese
I was able to build branch specific installers and these are now available for testing in the Downloads section of my fork. Please see: https://bitbucket.org/aloraine/integrated-genome-browser/downloads/.
Please note:
There are two commits on this branch. One affects the bitbucket pipelines configuration file only. The other affects the other configuration files (pom.xml files and intall4j configurations files.) This was done so that I can cherry-pick the bitbucket change onto other branches more easily. Changes to the poms will likely need to be done manually on those other branches.
Here again is the link to my new branch, ready for code review:
Harmonizing the code in our two repositories used to save jar-signing files for use in the bitbucket pipeline for building IGB installers. Making a new script that the pipeline will download and then run to set up the environment for the build. The goal here is to save IGB core team developers the effort of having to set up and configure environment variables in their repository. They will still have to configure their app password, the password used to enable copying of newly pipeline-built installers over to their repository's downloads section, a step necessary to facilitate initial testing.
Tested on Mac OS 12.5
Reset preferences to default and deleted the .igb folder.
I tested both the IGBF-3113.dmg and IGB-macos-9.1.10.dmg installers.
I was able to start IGB, load data, install ProtAnnot, open a genome, select genes, start ProtAnnot, stop ProtAnnot, and uninstall ProtAnnot without any issues or warnings/errors in the logs. I did not experience any issues related to the tab panels that I could identify.
I have modified the bitbucket pipeline configuration to avoid developer having to set up password and Install4J variables in their environment.
Now, what happens is that when you run the pipeline in your bitbucket IGB fork, a script will be downloaded into your build environment, which will be the new Docker container, as specified in the bitbucket configuration file. Next, a "source" command will run the script, which then downloads the assets the build processed needs. The build environment then builds IGB using mvn and the resulting files get copied to your repository's download directory in the usual way.
So now, the only repository variable you need to configure in order to get the pipeline to run is your "BB_AUTH_STRING". Instructions for what this needs to look like are in the top part of the Bitbucket pipelines YML file.
To test these changes, probably the easiest thing to do would be to add my IGB fork as a remote to your pwn personal fork. Then, pull my fork's branch "IGBF-3113" to your local clone and then push it to your fork.
If your fork is already set up to run bitbucket pipelines, then you probably already have repository variables set up, including the old Install4J license code, which will probably not work with the new Docker container, which has Install4J version 9.
So, if my changes are good, then the branch IGBF-3113 should build without any problems, because my new script will create environment variables that will over-ride your previously configure repository variables.
If the build works, you can then double-check that it works by simulating what a new developer would do. For that, you could delete all repository variables except your BB_AUTH_STRING, required for write access to your fork's Downloads folder.
If everything checks out with the above, then I will squash all the commits to the bitbucket pipelines file before submitting a PR. (I did the pipelines configuration in a few different steps which can be merged into one commit if you agree!)
To do:
- Update documentation - IGB Canvas course, Jira tickets ?
Tested by fetching IGBF-3113 branch from Dr. Loraine's fork of IGB
git fetch https://nfreese@bitbucket.org/aloraine/integrated-genome-browser.git IGBF-3113:IGBF-3113
I pushed to my BitBucket fork and Bitbucket pipelines worked correctly and the installer was created successfully.
I then removed all repository variables except BB_AUTH_STRING, BITBUCKET_PASSWORD, BITBUCKET_USERNAME, CODE_BASE_URL
The pipeline failed.
Failed pipeline log attached as nowlan_pipelineLog.txt
Thank you Nowlan Freese for testing. I appreciate it! I have changed the "env.sh" script to export the key passwords. Please try again - it should now work.
Pipeline failed.
Log attached as nowlan_pipelineLog2.txt
The error appears to be related to a failure of the JSON endpoint at UCSC:
[INFO] Plugin - Bookmarks and REST API .................... SUCCESS [ 1.318 s]
[INFO] Plugin - External View ............................. FAILURE [02:07 min]
Re-ran the pipeline and it worked this time.
Recommend pull request.
I tested the IGBF-3113.dmg installer from my branch and Dr. Loraine's branch on a Macbook Air with an M2 processor. Both installers asked to install Rosetta, which to my understanding should not occur with the new universal installers. After installing Rosetta, I was able to complete installation and run IGB successfully. While IGB was running I was able to confirm that it was running using Rosetta.
Could you try the same with this other installer - the one I made using my desktop installation of Install4J? Download it from https://bitbucket.org/aloraine/integrated-genome-browser/downloads/Zulu-IGB-macos-9.1.8.dmg.
Tested on Mac OS 12.5 with Intel processor.
Reset preferences to default and deleted the .igb folder.
Tested the master dmg installer.
I was able to start IGB, load data, install ProtAnnot, open a genome, select genes, start ProtAnnot, stop ProtAnnot, and uninstall ProtAnnot without any issues or warnings/errors in the logs.
Closing ticket.
Note that a new ticket has been created (IGBF-3158) to investigate the installer issue with Mac M1/2 processors.
Thank you for finding the problem. I modified the install4j configuration file to include:
architecture="universal"
attribute in tag "macosFolder"
I pushed the new commit directly to the master branch. New apple installer is built and ready for testing. Download as usual from https://bitbucket.org/lorainelab/integrated-genome-browser/downloads/.
attn: Nowlan Freese
UPDATE: The master branch no longer requires Rosetta when running on an M1 Mac.
Need to commit new Dockerfile and double-check that documentation is up-to-date with the changes.
New Docker file is merged from aloraine branch IGBF-3113 in repository https://bitbucket.org/lorainelab/integrated-genome-browser-docker.
Updating README for the Dockerfile repository at https://bitbucket.org/lorainelab/integrated-genome-browser-docker.
Installed desktop version of Install4J version 9 and upgraded to a new license for this new version. Added new license information to bitbucket repository where we keep this stuff. Added a new file to the IGB project: install4jv9.install4j. Decided not to use "xml" as the extension because the desktop version of install4j does not recognize this as a valid configuration file.