Implement Add Data Source (Quickload) endpoint

As mentioned in the linked issue, opening the "Preferences" window at the Data sources level worked when I created a custom endpoint for it. However, http request failed repeatedly yesterday. Turns out it was just a caching issue and after deleting .idea folder in the Intellij IGB project folder along with the IGB folder did the trick and the http request went through the right breakpoints during debugging and returned the correct response.

The idea to open the "Preferences" window or the "Add New Data Source" window was abandoned because it could only confuse the user. Instead the data source should be added directly to the "DataProviderManager" instance (where new data sources are added to). To do so the instances of the "DataProviderFactoryManager" and "DataProviderManager" is required. New instances of the classes may work as the setters of these classes are not used in the AddDataProvider class (extends JFrame) where the logic to add a new data source is from. However, so far this is has not worked.

DataProviderFactoryManager service is activated at the startup of the application and the factories are added (we need the Quickload factory). However, when the service is used the in IGBServiceImpl the factories are empty. Figuring out if the instance is emptied after use(logically unlikely) or the instance is the same in the handler.

For the solution above to work, a component class (DataProviderFactoryManger.class) object needs to be referenced from a non-component class (IgbServiceImpl.class). FrameworkUtil utility class can be used to get the bundle with which the service we need can be obtained. IgbServiceImpl uses this already and to get the DataProviderFactoryManager service this logic can be piggybacked on.
On implementation, however, the data source gets added (without using a thread, like in AddDataProvider.class) only after restarting IGB and not immediately. Threading should help here but it came with its own challenges, specifically, DataProviderManager instantiation is causing an issue.

The above solution has been implemented. To test, use the following request format:
http://127.0.0.1:7085/igbDataSource?quickloadurl= <URL>&quickloadname=<NAME>

Commit: https://bitbucket.org/KarthikRavee91/karthikfork-igb/commits/41906e2238a256e221093e0244e5455cf962f585

Tested on Mac using the following URL:

http://127.0.0.1:7085/igbDataSource?quickloadurl=https://data.cyverse.org/dav-anon/iplant/home/shared/BioViz/rnaseq&quickloadname=NOwlanTestT

I was able to add the Quickload source, load data, and remove the Quickload source without experiencing any issues or errors in the logs. I also tested adding the Quickload source when the Data Sources tab was open and did not experience any issues.

Changes have been implemented and pushed:
1) Change method name to appropriately reflect the action
2) Remove html response message and only return "OK" status

Pull Request: https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/900/igbf-3131

I was able to successfully use the endpoint to add data sources without encountering any issues. I have included some notes about testing below as I found some interesting edge cases that were unrelated to Karthik's changes.

Recommend pull request.

1. When I added the same quickload twice, e.g., hitting the new endpoint in exactly the same way more than once, there was no IGB modal informing me that the quickload had already been added and no indication in the IGB logs. This is OK behavior.
2. When I added an invalid quickload link, IGB throws a NullPointerException (see logs below). Strangely, this invalid Quickload is not highlighted in red in the Data Sources tab. However, this Quickload URL behaves the same in IGB release 9.1.8. This is an odd edge case that may be very specific to the URL used.
3. When I added an invalid quickload link that had a typo the IGB log showed that there was a problem reading contents.txt and the quickload was (correctly) highlighted in red in the Data Sources tab.
4. When I added a link to a CyVerse quickload site, there must have been an issue with CyVerse (momentarily, not consistent) that caused the site to hang. This caused IGB to hang for an extended period of time and I was unable to interact with the user interface or restart IGB. I had to delete the plist file in order to successfully start IGB (I assume because on startup IGB was attempting to reach the quickload). I do not think this is related to these changes, but is an issue I have seen from time to time where IGB will not start or is unresponsive if attempting to connect to an unresponsive quickload.

1) Valid quickload:
http://127.0.0.1:7085/igbDataSource?quickloadurl=https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.0&quickloadname=NOwlanTestT

Log:

15:53:51.163 INFO  o.l.i.q.QuickloadDataProvider - Initializing Quickload Server https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.0/
Jul 18, 2022 3:53:52 PM com.sun.webkit.network.PublicSuffixes lambda$static$1
WARNING: Resource not found: lib/security/public_suffix_list.dat
15:53:52.869 WARN  o.l.i.quickload.util.QuickloadUtils - Optional quickload synonyms.txt file could not be loaded from https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.0/synonyms.txt
15:53:53.177 WARN  o.l.i.quickload.util.QuickloadUtils - Optional species.txt could not be loaded from: https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/raw/release-v1.0.0/species.txt

2) Invalid quickload (incorrect URL where raw is replaced with src):
http://127.0.0.1:7085/igbDataSource?quickloadurl=https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/src/release-v1.0.0&quickloadname=NOwlanTestT-2

Log:

16:16:13.205 INFO  o.l.i.q.QuickloadDataProvider - Initializing Quickload Server https://bitbucket.org/nfreese/quickload-genome-in-a-bottle/src/release-v1.0.0/
16:16:14.132 INFO  c.a.igb.util.IGBTrustManager - Trusted Certificates:CN=bitbucket.org,O=Atlassian\, Inc.,L=San Francisco,ST=California,C=US,2.5.4.5=#130733393238343439,2.5.4.15=#0c1450726976617465204f7267616e697a6174696f6e,1.3.6.1.4.1.311.60.2.1.2=#130844656c6177617265,1.3.6.1.4.1.311.60.2.1.3=#13025553; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1,O=DigiCert Inc,C=US
16:16:14.532 INFO  c.a.igb.util.IGBTrustManager - Trusted Certificates:CN=bitbucket.org,O=Atlassian\, Inc.,L=San Francisco,ST=California,C=US,2.5.4.5=#130733393238343439,2.5.4.15=#0c1450726976617465204f7267616e697a6174696f6e,1.3.6.1.4.1.311.60.2.1.2=#130844656c6177617265,1.3.6.1.4.1.311.60.2.1.3=#13025553; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1,O=DigiCert Inc,C=US
16:16:15.650 INFO  c.a.igb.util.IGBTrustManager - Trusted Certificates:CN=bitbucket.org,O=Atlassian\, Inc.,L=San Francisco,ST=California,C=US,2.5.4.5=#130733393238343439,2.5.4.15=#0c1450726976617465204f7267616e697a6174696f6e,1.3.6.1.4.1.311.60.2.1.2=#130844656c6177617265,1.3.6.1.4.1.311.60.2.1.3=#13025553; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1,O=DigiCert Inc,C=US
16:16:15.978 INFO  c.a.igb.util.IGBTrustManager - Trusted Certificates:CN=bitbucket.org,O=Atlassian\, Inc.,L=San Francisco,ST=California,C=US,2.5.4.5=#130733393238343439,2.5.4.15=#0c1450726976617465204f7267616e697a6174696f6e,1.3.6.1.4.1.311.60.2.1.2=#130844656c6177617265,1.3.6.1.4.1.311.60.2.1.3=#13025553; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1,O=DigiCert Inc,C=US
16:16:17.037 INFO  c.a.igb.util.IGBTrustManager - Trusted Certificates:CN=bitbucket.org,O=Atlassian\, Inc.,L=San Francisco,ST=California,C=US,2.5.4.5=#130733393238343439,2.5.4.15=#0c1450726976617465204f7267616e697a6174696f6e,1.3.6.1.4.1.311.60.2.1.2=#130844656c6177617265,1.3.6.1.4.1.311.60.2.1.3=#13025553; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1,O=DigiCert Inc,C=US
16:16:17.535 INFO  c.a.igb.util.IGBTrustManager - Trusted Certificates:CN=bitbucket.org,O=Atlassian\, Inc.,L=San Francisco,ST=California,C=US,2.5.4.5=#130733393238343439,2.5.4.15=#0c1450726976617465204f7267616e697a6174696f6e,1.3.6.1.4.1.311.60.2.1.2=#130844656c6177617265,1.3.6.1.4.1.311.60.2.1.3=#13025553; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1,O=DigiCert Inc,C=US
Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException: genomeVersionNamePrefix is a required field.
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:229)
	at org.lorainelab.igb.synonymlookup.services.SpeciesInfo.<init>(SpeciesInfo.java:18)
	at org.lorainelab.igb.quickload.util.QuickloadUtils.lambda$parseSpeciesInfo$2(QuickloadUtils.java:93)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
	at org.lorainelab.igb.quickload.util.QuickloadUtils.parseSpeciesInfo(QuickloadUtils.java:83)
	at org.lorainelab.igb.quickload.util.QuickloadUtils.loadSpeciesInfo(QuickloadUtils.java:74)
	at org.lorainelab.igb.quickload.QuickloadDataProvider.loadOptionalQuickloadFiles(QuickloadDataProvider.java:130)
	at org.lorainelab.igb.quickload.QuickloadDataProvider.initialize(QuickloadDataProvider.java:114)
	at com.affymetrix.igb.general.DataProviderManager.initializeDataProvider(DataProviderManager.java:333)
	at com.affymetrix.igb.general.DataProviderManager.lambda$addDataProvider$205(DataProviderManager.java:324)
	at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
	at java.awt.EventQueue.access$500(EventQueue.java:97)
	at java.awt.EventQueue$3.run(EventQueue.java:709)
	at java.awt.EventQueue$3.run(EventQueue.java:703)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

3) Invalid quickload (typo):
http://127.0.0.1:7085/igbDataSource?quickloadurl=https://bitbucket.org/nfreese/quickload-gena-bottle/src/release-v1.0.0&quickloadname=NOwlanTestT-3

Log:

16:22:18.776 INFO  o.l.i.q.QuickloadDataProvider - Initializing Quickload Server https://bitbucket.org/nfreese/quickload-gena-bottle/src/release-v1.0.0/
16:22:19.038 WARN  o.l.i.quickload.util.QuickloadUtils - Could not read contents.txt from: https://bitbucket.org/nfreese/quickload-gena-bottle/src/release-v1.0.0/contents.txt
16:22:19.038 WARN  o.l.i.q.QuickloadDataProvider - Missing required quickload file, or could not reach source. This quickloak source will be disabled for this session.
16:22:19.242 WARN  o.l.i.quickload.util.QuickloadUtils - Optional quickload synonyms.txt file could not be loaded from https://bitbucket.org/nfreese/quickload-gena-bottle/src/release-v1.0.0/synonyms.txt
16:22:19.434 WARN  o.l.i.quickload.util.QuickloadUtils - Optional species.txt could not be loaded from: https://bitbucket.org/nfreese/quickload-gena-bottle/src/release-v1.0.0/species.txt

Changes requests for the PR:

Please provide developers documentation to explain how to "hit" this new endpoint and the expected behavior so that users can tell if it is working.

Documentation locations:

• A comment in this ticket
• Add javadoc to new entry in IgbServices.java interface
• Document addDataSourceToDataManagement using javadoc

Also, please change:

• Remove "glob" import, e.g., import java.awt.*

Please squash everything into one commit if at all possible.

I have come across an issue using Netbeans 14 where IGB fails to open with the IGBF-3131 branch changes.

The error in the IGB logs is:

ERROR: Bundle igb [31] EventDispatcher: Error during dispatch. (java.lang.NoClassDefFoundError: Could not initialize class com.affymetrix.igb.IgbServiceImpl)
java.lang.NoClassDefFoundError: Could not initialize class com.affymetrix.igb.IgbServiceImpl

There are two lines of code added in the commit to IgbServiceImpl.java where the issue occurs:

ServiceReference serviceRef = bundleContext.getServiceReference(DataProviderFactoryManager.class.getName());
dataProviderFactoryManager = (DataProviderFactoryManager) bundleContext.getService(serviceRef);

serviceRef is being assigned null which then causes the issue on the following line when it is used in bundleContext.getService(serviceRef).

So far we have only been able to replicate this issue on my Macbook Pro. Karthik does not experience the same issue on his Windows machine. The issue does appear to be independent of the JDK version used on my system. For additional testing, I have set up one of the Macbook Pros from the lab. I will update this ticket after we have carried out additional testing.

Added javadoc to the changes made. While squashing the commits, a few issues occurred so a new branch was created IGBF-3131a with all the changes pertaining to this issue and pushed. Commit

I'm still seeing the same error as before on the new branch IGBF-3131a using the Zulu JDK.

I was able to successfully add a quickload using the installer from Karthik's 3131a branch (IGBF-3131a.dmg). I will do some additional testing to see if the error I am experiencing in Netbeans is specific to Netbeans.

I was also able to successfully escape the ampersand and add a quickload using the endpoint with the following url:
http://127.0.0.1:7085/igbDataSource?quickloadurl=https://translate.bioviz.org/api/?hubUrl=https://bitbucket.org/nfreese/trackhub-human-hub/raw/main/hub.txt%26filePath=/&quickloadname=NowlanHG38

The issue I am experiencing seems very closely related to IGBF-1366. The error is only present when run in developer mode (see Ivory's comments in 1366).

Continuing investigation.

The error occurs when running from Netbeans as netbeans runs with:

-DdevelopmentMode=true

Removing the development mode flag and running the same command (see below) from the terminal opens IGB and the endpoint works correctly.

Wrapping the initializeMenuBarManagerServiceTracker() serviceRef line and dataProviderFactoryMaanger line in a try catch block allows IGB to start, but throws the following error: "java.lang.NullPointerException: Specified service reference cannot be null." Although IGB starts, the endpoint does not function.

Note that the following commands are specific to my (Nowlan's) machine.

#development mode - produces an error and IGB does not open

cd /Users/lorainelab/Documents/NowlanProjects/Repositories/nowlanfork-igb/main; JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu8.62.0.19-ca-fx-jdk8.0.332-macosx_x64/zulu-8.jdk/Contents/Home M2_HOME=/Users/lorainelab/utils/apache-maven-3.8.6 /Users/lorainelab/utils/apache-maven-3.8.6/bin/mvn -Dexec.executable=/Library/Java/JavaVirtualMachines/zulu8.62.0.19-ca-fx-jdk8.0.332-macosx_x64/zulu-8.jdk/Contents/Home/bin/java -Dexec.mainClass=com.affymetrix.main.Main "-Dexec.args=-Xmx1g -DdevelopmentMode=true -Dapple.laf.useScreenMenuBar=true -Dsun.java2d.opengl=true -classpath %classpath com.affymetrix.main.Main" "-Dexec.vmArgs=-Xmx1g -DdevelopmentMode=true -Dapple.laf.useScreenMenuBar=true -Dsun.java2d.opengl=true" -Dexec.appArgs= org.codehaus.mojo:exec-maven-plugin:1.2.1:exec

#non-development mode - IGB opens and the endpoint functions correctly

cd /Users/lorainelab/Documents/NowlanProjects/Repositories/nowlanfork-igb/main; JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu8.62.0.19-ca-fx-jdk8.0.332-macosx_x64/zulu-8.jdk/Contents/Home M2_HOME=/Users/lorainelab/utils/apache-maven-3.8.6 /Users/lorainelab/utils/apache-maven-3.8.6/bin/mvn -Dexec.executable=/Library/Java/JavaVirtualMachines/zulu8.62.0.19-ca-fx-jdk8.0.332-macosx_x64/zulu-8.jdk/Contents/Home/bin/java -Dexec.mainClass=com.affymetrix.main.Main "-Dexec.args=-Xmx1g -Dapple.laf.useScreenMenuBar=true -Dsun.java2d.opengl=true -classpath %classpath com.affymetrix.main.Main" "-Dexec.vmArgs=-Xmx1g -Dapple.laf.useScreenMenuBar=true -Dsun.java2d.opengl=true" -Dexec.appArgs= org.codehaus.mojo:exec-maven-plugin:1.2.1:exec

To resolve issues caused due to accessing DataProviderFactoryManager service reference before the bundle is activated in Mac OS, workflow is adjusted so that the reference is only accessed when the endpoint is hit and not at initialization of the bundle. The following commit was tested in Windows with Postman as well as Bioviz Public Trackhub page and works as intended. Commit

• Clean and build from Netbeans: worked with no issues.
• Run IGB in development mode from command line: worked with no issues.
• Download branch installer (.dmg), install and run: worked with no issues.

I did not encounter the previous error in any of the testing scenarios. IGB opened and was able to add the data sources each time without any issues.

Karthik Raveendran - can you double-check if the following import is required on line 14 of IgbServiceImpl.java:

import aQute.bnd.annotation.component.Reference;

Tested on Mac OS 12.5 with Netbeans 14.
Reset preferences to default and deleted the .igb folder.
Used this link for testing the data source endpoint: http://127.0.0.1:7085/igbDataSource?quickloadurl=https://translate.bioviz.org/api/?hubUrl=https://bitbucket.org/nfreese/trackhub-human-hub/raw/main/hub.txt%26filePath=/&quickloadname=NowlanHG38

No, Dr. Freese, that import is not required. Added that by accident.

Please remove the unneeded import and avoid use of "*" imports.

I think we should try to address the issue found in IGBF-3147 where the data source is not appearing in the Available Data window. Please also check that imports do not use * and squash the commits.

Add / create formal documentation to developers and/or users guide.
To start, Karthik to add text to here. Karthik to read through the Users Guide "Advanced Features" documentation and assess where documentation for the new feature (endpoint) should go.

https://wiki.bioviz.org/confluence/display/igbdevelopers/Bookmark+REST+API
https://wiki.bioviz.org/confluence/display/igbman/Controlling+IGB+using+IGB+Links+with+Genome+Synonyms

UPDATE: IGBF-3152 was created to add documentation regarding the various IGB endpoints, including this one.

PR Submitted https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/909/igbf-3131-add-data-source-endpoint

Merged.

Installers are built. Note: installers are using the old jdk with nonfunctional app manager. New installers will be made pending merge of IGBF-3113 changes.

Master branch is built using new Docker image and ready for testing.

Tested on MacOS 12.5 using master branch installer.
Reset preferences to default in IGB and removed .igb folder.

Able to successfully load data source through the endpoint using this URL: http://127.0.0.1:7085/igbDataSource?quickloadurl=https://data.cyverse.org/dav-anon/iplant/home/shared/BioViz/rnaseq&quickloadname=NOwlanTestT

Closing ticket.