Investigated the issue, it is happening because of the changes included in this commit: https://bitbucket.org/lorainelab/integrated-genome-browser/commits/a2c9ebd52cb1ccd4105d056b14ce86d89fe40114 when this commit is reverted, it is working properly, working on finding the exact issue and fixing it.

I'm also seeing an error when I try to load genomes provided by UCSC REST.

To reproduce:

1. Select Ovis aries in the Species dropdown
2. Select O_aries_Nov_2015 in the Genome Version dropdown
3. I see the following error in the log and no data is available under Available Data: ERROR - Couldn't find species for version O_aries_Nov_2015

Maybe this is due to some of the species name logic added for Ensembl? I will try some older commits and see where the error first starts and will update this comment.

UPDATE - The error I am seeing above also seems to be due to the IGBF-3780 commit. If I go back to the commit before the IGBF-3780 was merged into main then I do not see the above error.

After reverting the Ensembl code (branch: https://bitbucket.org/jaya-sravani/integrated-genome-browser/branch/IGBF-3853-testing-without-ensembl), the issue described in the ticket and the issue in the comment added by Nowlan Freese are not occurring, but the out of memory issue with Homo Sapiens species is still there (see the above comment), as this issue is happening because of the large number of datasets available (for the H_sapiens_Dec_2013 genome version there are around 9,500 datasets) for the Homo Sapiens species for UCSC Rest Provider, I checked it by manually limiting the size of the datasets to 500 and it's working perfectly. Right now, going through the code that is responsible for removing the datasets and trying to optimize it.

Found what is causing the out-of-memory issue, the below code in the ThreadUtils.java class creates a new thread for each dataset, ensuring tasks for the same object are executed one after another but possibly using more resources. As it's not reusing the threads it isn't possible to create a new thread if there are a lot of datasets.

public void execute(Object obj, CThreadWorker<?, ?> worker) {
      if (obj == null || worker == null) {
         throw new IllegalArgumentException("None of parameters can be null");
      }
     ThreadUtils.getPrimaryExecutor(obj).execute(worker);
}

public synchronized static Executor getPrimaryExecutor(Object key) {
     Executor exec = obj2exec.get(key);
     if (exec == null) {
        exec = Executors.newSingleThreadExecutor();
        obj2exec.put(key, exec);
     }
     return exec;
}

Changed this code to the below code which uses ExecutorService to manage the threads and their lifecycle, this doesn't give any error but the time taken to remove all the UCSC datasets (around 9,500) is around 4 minutes 30 seconds. That is because a few parts of code require sequential execution, if we want to optimize it, it might take a lot of effort, investigation, and a lot of code change.

public class ExecutorServiceManager {
    private static final int THREAD_POOL_SIZE = Runtime.getRuntime().availableProcessors();
    private static final ExecutorService executorService = Executors.newFixedThreadPool(THREAD_POOL_SIZE);

    public static ExecutorService getExecutorService() {
        return executorService;
    }
}
public void executeForRemoveDatasets(Object obj, CThreadWorker<?, ?> worker) {
    if (obj == null || worker == null) {
        throw new IllegalArgumentException("None of parameters can be null");
    }
    ExecutorServiceManager.getExecutorService().execute(worker);
}

As discussed with Nowlan Freese, the best option to resolve the issue at this point is to remove narrowPeak and bigWig file types, as they have a lot of datasets, and this will result in removing the datasets faster without any issues and takes far less time. Here is the updated branch: https://bitbucket.org/jaya-sravani/integrated-genome-browser/branch/IGBF-3853. Added it as a separate commit to make it more understandable.

Testing Sravani's branch on Mac.

• IGB built with tests.
• The bigWig and narrowPeak folders are no longer available under UCSC REST available data.
• Other UCSC REST data (RefSeq All) load correctly.
• Turning the UCSC REST Data Source off and back on works quickly.
• No issues in the log.

Ready for pull request.

Raised the Pull request: https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/1032

PR merged. New installers built and deployed to bioviz.org early access IGB section.

Tested using main branch installer on Mac.

• The bigWig and narrowPeak folders are no longer available under UCSC REST available data.
• Other UCSC REST data (RefSeq All) load correctly.
• Turning the UCSC REST Data Source off and back on works quickly.
• Able to load genomes provided by UCSC REST.
• No issues in the log.

Closing ticket.