Details
-
Type:
Bug
-
Status: In Progress (View Workflow)
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:None
-
Story Points:3
-
Epic Link:
-
Sprint:Spring 8, Spring 9
Description
Situation: A user reported an issue loading a local file in IGB 10.0.0. The issue appears to be that there was an illegal character, a space, in the name of a folder on the path to the file. The issue was not present in IGB 9.1.10. Therefore, the issue may be due to our upgrade in either the htsjdk or in Java.
The two filetypes affected appear to be bigwig and bigbed. Example files can be found here: https://data.bioviz.org/smokeTestingQuickload/H_sapiens_Dec_2013/
Task: Allow IGB to load bigwig and bigbed files from local paths that contain spaces.
Attachments
Issue Links
Activity
Attempting to build release-9.1.10 to compare to main branch. I'm seeing the following error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.8:copy (copy) on project shared-lib-wrapper: Unable to find artifact.: The following artifacts could not be resolved: org.lorainelab.igb:org.lorainelab.igb.error.reporter.api:jar:1.0.0 (absent): org.lorainelab.igb:org.lorainelab.igb.error.reporter.api:jar:1.0.0 was not found in https://maven.bioviz.org/repo1
Ann Loraine - did we move the artifact org.lorainelab.igb.error.reporter.api out of repo1?
BigWigSymLoader.java in initbbReader is making a call to SeekableStreamFactory in the HTSJDK. We didn't make any changes to BigWigSymLoader as part of the upgrade to JDK21. However, there was a commit in 2020 that changed how the handling of file paths occurred in SeekableStreamFactory.java.
// convert to URI in order to obtain a decoded version of the path string suitable // for use with the File constructor final String decodedPath = new URI(path).getPath(); return new SeekableFileStream(new File(decodedPath));
else if (path.startsWith("file:")) { return new SeekableFileStream(new File(new URL(path).getPath()));
It looks like the SeekableStreamFactory.java code is changed again in the most recent commit.
We may just need to re-encode the URI before we send it to the SeekableStreamFactory (works in debug mode). However, it is unclear what the most recent changes to the HTSJDK will expect, so we will need to test this again after Kaushik's changes to the HTSJDK.
From initial testing it appears that only bigWig and bigBed were affected, but may need to double-check. I'm curious as to how/if we are using other filetypes with the SeekableStreamFactory.
UPDATE - It looks like the main diff in the 2020 vs 2016 HTSJDK code is changing from a Java URL to URI. It looks like URI does not encode spaces the same way that URL does, which matches with the issue we are seeing. See this stackoverflow for more info.
UPDATE 2 - Our code (BigWigSymLoader) was using a URI, but because the 2016 HTSJDK was expecting a URL we were using GeneralUtils.URLDecode on the URI. This seems to be the issue.
UPDATE 3 - See IGBF-1471 and this commit for the last time we made a change to the above logic.
Update on build error:
- I have confirmed on my fork that release-9.1.10 cannot build; I observed the same error reported by Nowlan Freese
- In between release of IGB 9.1.10 and IGB 10.0.0, we moved our "legacy" maven repositories to a new host. In the process I guess that artifact was lost?
- I have re-started the old host with the old data deployed. I configured the firewall to allow login from any IP address. This is so that we can retrieve the missing artifact if required.
- I am looking on the old host for artifact: org.lorainelab.igb:org.lorainelab.igb.error.reporter.api:jar:1.0.0
The artifact is here:
maven.bioviz.org $ pwd
/var/www/html
maven.bioviz.org $ find . | grep .jar | grep org.lorainelab.igb.error.reporter.api
./repo2/org/lorainelab/igb/org.lorainelab.igb.error.reporter.api/1.0.0/org.lorainelab.igb.error.reporter.api-1.0.0.jar
./repo2/org/lorainelab/igb/org.lorainelab.igb.error.reporter.api/1.0.0/org.lorainelab.igb.error.reporter.api-1.0.0.jar.sha1
./repo2/org/lorainelab/igb/org.lorainelab.igb.error.reporter.api/1.0.0/org.lorainelab.igb.error.reporter.api-1.0.0.jar.md5
To solve this problem, we will likely need to deploy multiple artifacts. I will work on it Nowlan Freese.
Packages we will likely need to build IGB 9:
maven.bioviz.org $ ls org.lorainelab.igb.remote.* org.lorainelab.igb.remote.services.commons: 1.0.0 maven-metadata.xml maven-metadata.xml.md5 maven-metadata.xml.sha1 org.lorainelab.igb.remote.services.discovery.provider.configured: 1.0.0 maven-metadata.xml maven-metadata.xml.md5 maven-metadata.xml.sha1 org.lorainelab.igb.remote.services.distribution.provider.http: 1.0.0 maven-metadata.xml maven-metadata.xml.md5 maven-metadata.xml.sha1 org.lorainelab.igb.remote.services.parent: 1.0.0 maven-metadata.xml maven-metadata.xml.md5 maven-metadata.xml.sha1 org.lorainelab.igb.remote.services.topology.manager: 1.0.0 maven-metadata.xml maven-metadata.xml.md5 maven-metadata.xml.sha1 maven.bioviz.org $ ls org.lorainelab.igb.remote.* org.lorainelab.igb.remote.services.commons: 1.0.0 maven-metadata.xml maven-metadata.xml.md5 maven-metadata.xml.sha1 org.lorainelab.igb.remote.services.discovery.provider.configured: 1.0.0 maven-metadata.xml maven-metadata.xml.md5 maven-metadata.xml.sha1 org.lorainelab.igb.remote.services.distribution.provider.http: 1.0.0 maven-metadata.xml maven-metadata.xml.md5 maven-metadata.xml.sha1 org.lorainelab.igb.remote.services.parent: 1.0.0 maven-metadata.xml maven-metadata.xml.md5 maven-metadata.xml.sha1 org.lorainelab.igb.remote.services.topology.manager: 1.0.0 maven-metadata.xml maven-metadata.xml.md5 maven-metadata.xml.sha1
Update:
- I deployed the missing "error reporting" artifacts to "repo2" on our newer legacy maven repository host available at maven.bioviz.org
- I confirmed that my fork can build release-9.1.10
Thank you Nowlan Freese for bringing this error to my attention.
Error: