Problem was due to an expired jar-signing key from Apple.

Created new Apple developer key for signing jar files.

Progress:

• Edited bitbucket_pipelines.yml and Install4J configuration file to use new keyfile for signing Jars.
• Pushed edits to branch release-9.1.4 on fork: https://bitbucket.org/aloraine/integrated-genome-browser/branch/release-9.1.4
• Pushed the new keyfile to the private repository where we store assets used to build installers.

However, am getting an error when I run "release" pipeline.
This is the error:
[INFO] — install4j-maven-plugin:1.1.1:compile (compile-installers) @ main —
[INFO] install4j version 7.0.8 (build 7248), built on 2018-11-06
[INFO] Using Java 1.8.0_212 from /usr/lib/jvm/jdk1.8.0_212/jre
[INFO] Registered to
[INFO]
[INFO] Loading config file /opt/atlassian/pipelines/agent/build/distribution/igb_7.install4j
[INFO] install4j: compilation failed. Reason: Password might be wrong. (macOS)

I checked the password on my local (Mac) using the following command:

• keytool -list -v -storetype pkcs12 -keystore 2020-03-25-developerID_application_igb.p12

The above command asks for a password. If I supply the right one, I get a lot of information printed about the file. If I supply the wrong password, I get this error:

local aloraine$ keytool -list -v -storetype pkcs12 -keystore 2020-03-25-developerID_application_igb.p12
Enter keystore password:
keytool error: java.io.IOException: keystore password was incorrect
java.io.IOException: keystore password was incorrect
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059)
at java.security.KeyStore.load(KeyStore.java:1445)
at sun.security.tools.keytool.Main.doCommands(Main.java:926)
at sun.security.tools.keytool.Main.run(Main.java:366)
at sun.security.tools.keytool.Main.main(Main.java:359)
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
... 5 more

I am using the correct password. It is the same password as the one we previously used. So I am not sure why this is failing.

Inspecting old key. Note expiration date:

keytool -list -v -storetype pkcs12 -keystore developerID_application_igb.p12 > oldkey.out

oldkey.out:

Keystore type: PKCS12
Keystore provider: SunJSSE
... [deleted stuff]
Owner: C=US, O=Ann Loraine, OU=V42Q2N95L9, CN=Developer ID Application: Ann Loraine (V42Q2N95L9), UID=V42Q2N95L9
Issuer: C=US, O=Apple Inc., OU=Apple Certification Authority, CN=Developer ID Certification Authority
... [deleted stuff]
Valid from: Thu Jan 15 15:49:00 EST 2015 until: Thu Jan 16 15:49:00 EST 2020
... [deleted stuff]
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
... [deleted stuff]

Prof. [~aloraine], we have tried following things:
1) Added your repo as an upstream to my fork.
2) Pulled the release-9.1.4 branch from there and pushed it to my fork.
3) Created a fork of IIPC and renamed new key to the old one and deleted the old key.
4) Changed bb pipeline configuration file to reflect the changes related to reference of the key and set it to my fork.
5) Built pipeline on my IGB fork.

Still Pooja Nikhare and I are facing the same issue. We came to a conclusion that there is a problem with the key itself and not the configuration part because when we change the references to the old key then the pipeline is building correctly.
Could you please generate the key again? Also, we wanted to know the detailed steps for key generation.

Update:
We tried to replicate the configuration by installing install4j s/w on my Mac and built the project. It has successfully generated the mac installer with new key locally. We are confused right now if its because of the docker image or the bitbucket pipeline configuration.
Steps for generating installer locally:
1) Install install4j version 7 on your machine.
2) Copy the install4j license from https://bitbucket.org/lorainelab/igb-installer-private-content/src/master/bbpipelines_environment_variables.txt
3) Open install4j app and click on add key.
4) Paste the key which was copied earlier.
5) install4j console will be opened. Now, click on open project button which is present on uppermost left corner of the console.
6) select the igb_7.install4j file from distribution folder in the local igb repository folder and open it.
7) Go to https://bitbucket.org/lorainelab/igb-installer-private-content/src/master/ repo and download the new key "2020-03-25-developerID_application_igb.p12" and paste it to the same distribution folder of local igb repo.
8) Under general setting in install4j app console go to code signing and uncheck windows option since we just want to check mac installer.
9) Now, click the build project button which is present in the same line as that of open project button.
10) It will prompt you to enter mac store password which can be entered directly. It is also present in the https://bitbucket.org/lorainelab/igb-installer-private-content/src/master/bbpipelines_environment_variables.txt file.
11) Once the build is successful, go to distribution folder again and in that directory go to build folder.
12) You would be able to see a .dmg file which can be installed on your mac.

One thing I have noticed about the key:

local aloraine$ ls -lh *.p12
rw-rr-@ 1 aloraine staff 3.1K Mar 25 12:06 2020-03-25-developerID_application_igb.p12
rw-rr- 1 aloraine staff 2.9K Oct 5 11:51 developerID_application_igb.p12

On my Mac system, there is a "@" symbol next to the file permissions string for the new key. This seems weird. Could this be causing a problem in the Docker image but not on Prutha's Mac computer?

As you know, an ssh client refuses to use private keys that do not have the correct file permissions. If the key is readable or write-able by "other" or "group" users, ssh refuses to use it.

I do not now what the "@" symbol means here.

Here is what I found on internet:
@ signifies that the file has extended attributes. Those attributes are usually used to signify that the file came from a package, was downloaded from the internet, etc.

ls -al@ imap.a
will show you the extended attributes that are saved for that file.

We are trying to simulate the pipeline environment on the local.
Pooja Nikhare and I were working on copying the igb project in the docker image and building it inside the docker image using COPY and RUN commands in the docker file. Once that is done, we can simply use docker run command to run the docker image. Since I am working half day today, Pooja Nikhare will continue to work on the same.

Updates :
I and Prutha Kulkarni tried to simulate the pipeline environment locally.
Steps :

• Fork and clone the repo - https://bitbucket.org/lorainelab/integrated-genome-browser-docker/src/master/
• Install the docker application locally
• Make changes to DockerFile inside the repo , to copy the project inside docker and to build it using following commands
  WORKDIR usr
  COPY integrated-genome-browser src/
  WORKDIR src
  RUN mvn install -P release-bitbucket-pipelines -Dinstall4j.licenseKey=$ {INSTALL4J_LICENSE}

  -Dinstall4j.winKeystorePassword=$

  {WIN_KEYSTORE_PASSWORD}

  -Dinstall4j.macKeystorePassword=$

  {MAC_KEYSTORE_PASSWORD}

• Edit license key, mac , windows keystorepassword according to variables given in https://bitbucket.org/lorainelab/igb-installer-private-content/src/master/
• build docker image using ' docker build . '
• run docker image : ' docker run <imageId>
• verify whether installers got created inside src/distribution/build folder
• copied the mac installer using command:
  docker cp <containerId>:/usr/src/distribution/build/IGB-macos-9.1.4.dmg <destAddress>
• Tested the installer on Mac PC.
  Prof [~aloraine] - If possible could you please change the file permissions, and then we can test again.

Please clarify:

• When you built the Mac IGB installer on a Mac, could the installer be used to install IGB on the same machine?
• When you built the Mac IGB installer using the Docker image running on a local host, could the installer then be used to install IGB on a Mac?
• How should the permissions for the key be changed?

[~aloraine]
When you built the Mac IGB installer on a Mac, could the installer be used to install IGB on the same machine? Yes It can be used to install IGB on the same machine.
When you built the Mac IGB installer using the Docker image running on a local host, could the installer then be used to install IGB on a Mac? Yes the installer which was built using the docker image on the local can be used to install IGB on mac.
How should the permissions for the key be changed? For this, I am not sure if the permission of the key itself is causing the problem. As we have used the same key in both scenarios to build the installer. Still if you want to change the attributes then you could use following command:
xattr -c <File_Name i.e. name of the mac key file>
As @ means the file has some extended attributes the -c flag will clear those extended attributes from the permissions of the file.

Thank you for the detailed explanation Prutha Kulkarni.
Based on the findings of yourself and Pooja Nikhare, the permissions surely cannot be the problem.
To further check, I cloned a fresh copy of the repo containing the key. The key file in the newly cloned repo lacks the "@" (extended attributes) file permissions character. Therefore this cannot be the problem because a fresh copy of the key is obtained on every bitbucket build.

Please note: Since last week, one of the IGB tests has begun failing due to a Web resource no longer being available. Srishti Tiwari has committed a fix for same, which is now merged to the master branch. I have cherry-picked her commit and added it to release-9.1.4 on my fork. I re-ran the release's bitbucket pipeline after pushing the cherry-picked commit to my fork. The same error occurs:

"[INFO] — install4j-maven-plugin:1.1.1:compile (compile-installers) @ main —
[INFO] install4j version 7.0.8 (build 7248), built on 2018-11-06
[INFO] Using Java 1.8.0_212 from /usr/lib/jvm/jdk1.8.0_212/jre
[INFO] Registered to
[INFO]
[INFO] Loading config file /opt/atlassian/pipelines/agent/build/distribution/igb_7.install4j
[INFO] install4j: compilation failed. Reason: Password might be wrong. (macOS)
[INFO] ------------------------------------------------------------------------"

Prutha Kulkarni and Pooja Nikhare:

For the next steps, let's try the tried-and-true tactic of googling the error message Unfortunately, this is the only one of two ideas I have at this time.

The other idea:

I think there is something going wrong somehow with the bitbucket environment since Pooja Nikhare and Prutha Kulkarni have been able to successfully build a non-defective MacOS installers using a Mac computer (Prutha) and our Docker image (Pooja) running on a local host.

For the next attempt, I think we should do a deeper dive into the jar-signing step. Probably Install4J is literally just using the jar-signing utility from the java distribution itself, which is packaged with our Docker image. Since that is the step that seems to be failing, we ought to try directly calling that step within the bitbucket environment itself by adding a jar-signing call to the pipeline YML file and observing the outputs to stdout and stderr.

This also may be useful:
https://www.ej-technologies.com/resources/install4j/help/doc/concepts/codeSigning.html

Release branch release-9.1.4 is now updated:

• configured to use new Mac jar-signing file
• includes cherry-picked commit to halt testing against defunct site biodas.org

cc: Pooja Nikhare Prutha Kulkarni

Progress :

• We forked the repo with changes for the release branch release-9.1.4.
• set codesigning for macEnabled as false in igb_7.install4j and triggered the pipeline.
• The pipeline for release-9.1-4 branch builds the installers.
• And Prutha Kulkarni could install the .dmg file from the download section.
  We are still working on it and trying to debug the issue.

Update :
Current bitbucket-pipelines.yml config :

• curl -s -S -L -O https://$ {BB_AUTH_STRING}@api.bitbucket.org/2.0/repositories/lorainelab/iipc/src/master/2020-03-25-developerID_application_igb.p12
  
  Where 'iipc' is a forked repo of 'IGB Installer Private Content' and which does not contain the new mac certificate.
  (Note : The curl command creates a new file even if the file doesn't exist in the repo, so we never got any error like 'file does not exist error')
  
  Prof. [~aloraine] We need to change in the bitbucket-pipelines.yml config file :
  iipc to igb-installer-private-content as following or add new macOs cert file in the iipc repository
  - curl -s -S -L -O https://${BB_AUTH_STRING}

  @api.bitbucket.org/2.0/repositories/lorainelab/igb-installer-private-content/src/master/2020-03-25-developerID_application_igb.p12

Fork is now up-to-date.
cc: Pooja Nikhare Prutha Kulkarni

Release installer built on team repository following update of fork.
cc: Pooja Nikhare and Prutha Kulkarni

[~aloraine] Pooja Nikhare
I am able to install the recently built installer on the mac now.

Marking this issue as done.
Thank you Pooja Nikhare and Prutha Kulkarni for your step-by-step notes.
What you have written above will help greatly with future efforts.