Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-442

Implement Caching solution for local file storage to speed up data access of commonly accessed files

    Details

      Description

      replacing LocalUrlCacher would be a very good thing, but this initial implementation does not have to be a complete replacement immediately for the existing caching solution.

      Infinispan is a mature caching solution which should be easy to leverage to get a much smarter more effective file cache than our home rolled LocalUrlCacher.

        Attachments

          Activity

          Hide
          dcnorris David Norris (Inactive) added a comment -

          Discuss with John how to test this new feature.

          Show
          dcnorris David Norris (Inactive) added a comment - Discuss with John how to test this new feature.
          Hide
          nfreese Nowlan Freese added a comment -

          Several attendees of the IGB workshop at the Society for Developmental Biology expressed interest in being able to cache both the genome and annotation files for major model organisms. One researcher commented that he only works with human, and would love to have it locally cached so that he doesn't have to wait as long every time he needs to load sequence.

          Show
          nfreese Nowlan Freese added a comment - Several attendees of the IGB workshop at the Society for Developmental Biology expressed interest in being able to cache both the genome and annotation files for major model organisms. One researcher commented that he only works with human, and would love to have it locally cached so that he doesn't have to wait as long every time he needs to load sequence.
          Hide
          mason Mason Meyer (Inactive) added a comment - - edited

          I noticed several exceptions on my Windows machine as soon as I select a genome in IGB:

          13:30:15.459 ERROR c.l.c.d.RemoteFileDiskCacheService - Error downloading: Failed to delete original file 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3o=\cache\data_c2357382-27ae-4ee7-b9ab-75af8e874d09.tmp' after copy to 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3o=\cache\data.dat'
          java.io.IOException: Failed to delete original file 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3o=\cache\data_c2357382-27ae-4ee7-b9ab-75af8e874d09.tmp' after copy to 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3o=\cache\data.dat'
          at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2835) ~[commons-io-2.4.jar:2.4]
          at com.lorainelab.cache.disk.RemoteFileDiskCacheService.tryDownload(RemoteFileDiskCacheService.java:418) [cache-8.5.0.jar:na]
          at com.lorainelab.cache.disk.RemoteFileDiskCacheService.access$700(RemoteFileDiskCacheService.java:69) [cache-8.5.0.jar:na]
          at com.lorainelab.cache.disk.RemoteFileDiskCacheService$1.runInBackground(RemoteFileDiskCacheService.java:230) [cache-8.5.0.jar:na]
          at com.lorainelab.cache.disk.RemoteFileDiskCacheService$1.runInBackground(RemoteFileDiskCacheService.java:201) [cache-8.5.0.jar:na]
          at com.affymetrix.genometry.thread.CThreadWorker.doInBackground(CThreadWorker.java:73) [genometry-8.5.0.jar:na]
          at javax.swing.SwingWorker$1.call(Unknown Source) [na:1.8.0_40]
          at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_40]
          at javax.swing.SwingWorker.run(Unknown Source) [na:1.8.0_40]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_40]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_40]
          at java.lang.Thread.run(Unknown Source) [na:1.8.0_40]
          13:30:15.504 INFO c.a.genometry.util.ServerUtils - STOPWATCH METRICS for createSymLoader 717.9 ms
          13:30:15.621 ERROR c.l.c.d.RemoteFileDiskCacheService - Error downloading: Failed to delete original file 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3oudGJp\cache\data_f0ef5b2e-09b7-4ab3-944d-87396463b85d.tmp' after copy to 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3oudGJp\cache\data.dat'
          java.io.IOException: Failed to delete original file 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3oudGJp\cache\data_f0ef5b2e-09b7-4ab3-944d-87396463b85d.tmp' after copy to 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3oudGJp\cache\data.dat'
          at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2835) ~[commons-io-2.4.jar:2.4]
          at com.lorainelab.cache.disk.RemoteFileDiskCacheService.tryDownload(RemoteFileDiskCacheService.java:418) [cache-8.5.0.jar:na]
          at com.lorainelab.cache.disk.RemoteFileDiskCacheService.access$700(RemoteFileDiskCacheService.java:69) [cache-8.5.0.jar:na]
          at com.lorainelab.cache.disk.RemoteFileDiskCacheService$1.runInBackground(RemoteFileDiskCacheService.java:230) [cache-8.5.0.jar:na]
          at com.lorainelab.cache.disk.RemoteFileDiskCacheService$1.runInBackground(RemoteFileDiskCacheService.java:201) [cache-8.5.0.jar:na]
          at com.affymetrix.genometry.thread.CThreadWorker.doInBackground(CThreadWorker.java:73) [genometry-8.5.0.jar:na]
          at javax.swing.SwingWorker$1.call(Unknown Source) [na:1.8.0_40]
          at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_40]
          at javax.swing.SwingWorker.run(Unknown Source) [na:1.8.0_40]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_40]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_40]
          at java.lang.Thread.run(Unknown Source) [na:1.8.0_40]

          This exception repeats several times as it does above.

          Show
          mason Mason Meyer (Inactive) added a comment - - edited I noticed several exceptions on my Windows machine as soon as I select a genome in IGB: 13:30:15.459 ERROR c.l.c.d.RemoteFileDiskCacheService - Error downloading: Failed to delete original file 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3o=\cache\data_c2357382-27ae-4ee7-b9ab-75af8e874d09.tmp' after copy to 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3o=\cache\data.dat' java.io.IOException: Failed to delete original file 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3o=\cache\data_c2357382-27ae-4ee7-b9ab-75af8e874d09.tmp' after copy to 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3o=\cache\data.dat' at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2835) ~ [commons-io-2.4.jar:2.4] at com.lorainelab.cache.disk.RemoteFileDiskCacheService.tryDownload(RemoteFileDiskCacheService.java:418) [cache-8.5.0.jar:na] at com.lorainelab.cache.disk.RemoteFileDiskCacheService.access$700(RemoteFileDiskCacheService.java:69) [cache-8.5.0.jar:na] at com.lorainelab.cache.disk.RemoteFileDiskCacheService$1.runInBackground(RemoteFileDiskCacheService.java:230) [cache-8.5.0.jar:na] at com.lorainelab.cache.disk.RemoteFileDiskCacheService$1.runInBackground(RemoteFileDiskCacheService.java:201) [cache-8.5.0.jar:na] at com.affymetrix.genometry.thread.CThreadWorker.doInBackground(CThreadWorker.java:73) [genometry-8.5.0.jar:na] at javax.swing.SwingWorker$1.call(Unknown Source) [na:1.8.0_40] at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_40] at javax.swing.SwingWorker.run(Unknown Source) [na:1.8.0_40] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_40] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_40] at java.lang.Thread.run(Unknown Source) [na:1.8.0_40] 13:30:15.504 INFO c.a.genometry.util.ServerUtils - STOPWATCH METRICS for createSymLoader 717.9 ms 13:30:15.621 ERROR c.l.c.d.RemoteFileDiskCacheService - Error downloading: Failed to delete original file 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3oudGJp\cache\data_f0ef5b2e-09b7-4ab3-944d-87396463b85d.tmp' after copy to 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3oudGJp\cache\data.dat' java.io.IOException: Failed to delete original file 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3oudGJp\cache\data_f0ef5b2e-09b7-4ab3-944d-87396463b85d.tmp' after copy to 'C:\Users\lorainelab\AppData\Roaming\IGB\fileCache\aWdicXVpY2tsb2FkLm9yZy9xdWlja2xvYWQvQV90aGFsaWFuYV9KdW5fMjAwOS9UQUlSMTBfbVJOQS5iZWQuZ3oudGJp\cache\data.dat' at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2835) ~ [commons-io-2.4.jar:2.4] at com.lorainelab.cache.disk.RemoteFileDiskCacheService.tryDownload(RemoteFileDiskCacheService.java:418) [cache-8.5.0.jar:na] at com.lorainelab.cache.disk.RemoteFileDiskCacheService.access$700(RemoteFileDiskCacheService.java:69) [cache-8.5.0.jar:na] at com.lorainelab.cache.disk.RemoteFileDiskCacheService$1.runInBackground(RemoteFileDiskCacheService.java:230) [cache-8.5.0.jar:na] at com.lorainelab.cache.disk.RemoteFileDiskCacheService$1.runInBackground(RemoteFileDiskCacheService.java:201) [cache-8.5.0.jar:na] at com.affymetrix.genometry.thread.CThreadWorker.doInBackground(CThreadWorker.java:73) [genometry-8.5.0.jar:na] at javax.swing.SwingWorker$1.call(Unknown Source) [na:1.8.0_40] at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_40] at javax.swing.SwingWorker.run(Unknown Source) [na:1.8.0_40] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_40] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_40] at java.lang.Thread.run(Unknown Source) [na:1.8.0_40] This exception repeats several times as it does above.
          Hide
          mason Mason Meyer (Inactive) added a comment - - edited

          I noticed a null pointer exception in Windows when clicking on the Cache tab in Preferences.
          12:46:41.281 ERROR c.l.c.c.panel.CacheTableModel - null
          java.net.MalformedURLException: null
          at java.net.URL.<init>(Unknown Source) ~[na:1.8.0_45]
          at java.net.URL.<init>(Unknown Source) ~[na:1.8.0_45]
          at java.net.URL.<init>(Unknown Source) ~[na:1.8.0_45]
          at com.lorainelab.cache.configuration.panel.CacheTableModel.getValueAt(CacheTableModel.java:88) ~[cache-8.5.0.jar:na]
          at javax.swing.JTable.getValueAt(Unknown Source) [na:1.8.0_45]
          at javax.swing.JTable.prepareRenderer(Unknown Source) [na:1.8.0_45]
          at com.lorainelab.cache.configuration.panel.CacheConfigurationPanel$3.prepareRenderer(CacheConfigurationPanel.java:323) [cache-8.5.0.jar:na]
          at javax.swing.plaf.basic.BasicTableUI.paintCell(Unknown Source) [na:1.8.0_45]
          at javax.swing.plaf.basic.BasicTableUI.paintCells(Unknown Source) [na:1.8.0_45]
          at javax.swing.plaf.basic.BasicTableUI.paint(Unknown Source) [na:1.8.0_45]
          at javax.swing.plaf.ComponentUI.update(Unknown Source) [na:1.8.0_45]
          at javax.swing.JComponent.paintComponent(Unknown Source) [na:1.8.0_45]
          at javax.swing.JComponent.paint(Unknown Source) [na:1.8.0_45]
          at javax.swing.JComponent.paintChildren(Unknown Source) [na:1.8.0_45]
          at javax.swing.JComponent.paint(Unknown Source) [na:1.8.0_45]
          at javax.swing.JViewport.paint(Unknown Source) [na:1.8.0_45]
          at javax.swing.JComponent.paintChildren(Unknown Source) [na:1.8.0_45]
          at javax.swing.JComponent.paint(Unknown Source) [na:1.8.0_45]
          at javax.swing.JComponent.paintToOffscreen(Unknown Source) [na:1.8.0_45]
          at javax.swing.BufferStrategyPaintManager.paint(Unknown Source) [na:1.8.0_45]
          at javax.swing.RepaintManager.paint(Unknown Source) [na:1.8.0_45]
          at javax.swing.JComponent._paintImmediately(Unknown Source) [na:1.8.0_45]
          at javax.swing.JComponent.paintImmediately(Unknown Source) [na:1.8.0_45]
          at javax.swing.RepaintManager$4.run(Unknown Source) [na:1.8.0_45]
          at javax.swing.RepaintManager$4.run(Unknown Source) [na:1.8.0_45]
          at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_45]
          at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source) [na:1.8.0_45]
          at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source) [na:1.8.0_45]
          at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source) [na:1.8.0_45]
          at javax.swing.RepaintManager.prePaintDirtyRegions(Unknown Source) [na:1.8.0_45]
          at javax.swing.RepaintManager.access$1300(Unknown Source) [na:1.8.0_45]
          at javax.swing.RepaintManager$ProcessingRunnable.run(Unknown Source) [na:1.8.0_45]
          at java.awt.event.InvocationEvent.dispatch(Unknown Source) [na:1.8.0_45]
          at java.awt.EventQueue.dispatchEventImpl(Unknown Source) [na:1.8.0_45]
          at java.awt.EventQueue.access$500(Unknown Source) [na:1.8.0_45]
          at java.awt.EventQueue$3.run(Unknown Source) [na:1.8.0_45]
          at java.awt.EventQueue$3.run(Unknown Source) [na:1.8.0_45]
          at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_45]
          at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source) [na:1.8.0_45]
          at java.awt.EventQueue.dispatchEvent(Unknown Source) [na:1.8.0_45]
          at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source) [na:1.8.0_45]
          at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source) [na:1.8.0_45]
          at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source) [na:1.8.0_45]
          at java.awt.EventDispatchThread.pumpEvents(Unknown Source) [na:1.8.0_45]
          at java.awt.EventDispatchThread.pumpEvents(Unknown Source) [na:1.8.0_45]
          at java.awt.EventDispatchThread.run(Unknown Source) [na:1.8.0_45]
          Caused by: java.lang.NullPointerException: null
          ... 46 common frames omitted

          Show
          mason Mason Meyer (Inactive) added a comment - - edited I noticed a null pointer exception in Windows when clicking on the Cache tab in Preferences. 12:46:41.281 ERROR c.l.c.c.panel.CacheTableModel - null java.net.MalformedURLException: null at java.net.URL.<init>(Unknown Source) ~ [na:1.8.0_45] at java.net.URL.<init>(Unknown Source) ~ [na:1.8.0_45] at java.net.URL.<init>(Unknown Source) ~ [na:1.8.0_45] at com.lorainelab.cache.configuration.panel.CacheTableModel.getValueAt(CacheTableModel.java:88) ~ [cache-8.5.0.jar:na] at javax.swing.JTable.getValueAt(Unknown Source) [na:1.8.0_45] at javax.swing.JTable.prepareRenderer(Unknown Source) [na:1.8.0_45] at com.lorainelab.cache.configuration.panel.CacheConfigurationPanel$3.prepareRenderer(CacheConfigurationPanel.java:323) [cache-8.5.0.jar:na] at javax.swing.plaf.basic.BasicTableUI.paintCell(Unknown Source) [na:1.8.0_45] at javax.swing.plaf.basic.BasicTableUI.paintCells(Unknown Source) [na:1.8.0_45] at javax.swing.plaf.basic.BasicTableUI.paint(Unknown Source) [na:1.8.0_45] at javax.swing.plaf.ComponentUI.update(Unknown Source) [na:1.8.0_45] at javax.swing.JComponent.paintComponent(Unknown Source) [na:1.8.0_45] at javax.swing.JComponent.paint(Unknown Source) [na:1.8.0_45] at javax.swing.JComponent.paintChildren(Unknown Source) [na:1.8.0_45] at javax.swing.JComponent.paint(Unknown Source) [na:1.8.0_45] at javax.swing.JViewport.paint(Unknown Source) [na:1.8.0_45] at javax.swing.JComponent.paintChildren(Unknown Source) [na:1.8.0_45] at javax.swing.JComponent.paint(Unknown Source) [na:1.8.0_45] at javax.swing.JComponent.paintToOffscreen(Unknown Source) [na:1.8.0_45] at javax.swing.BufferStrategyPaintManager.paint(Unknown Source) [na:1.8.0_45] at javax.swing.RepaintManager.paint(Unknown Source) [na:1.8.0_45] at javax.swing.JComponent._paintImmediately(Unknown Source) [na:1.8.0_45] at javax.swing.JComponent.paintImmediately(Unknown Source) [na:1.8.0_45] at javax.swing.RepaintManager$4.run(Unknown Source) [na:1.8.0_45] at javax.swing.RepaintManager$4.run(Unknown Source) [na:1.8.0_45] at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_45] at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source) [na:1.8.0_45] at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source) [na:1.8.0_45] at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source) [na:1.8.0_45] at javax.swing.RepaintManager.prePaintDirtyRegions(Unknown Source) [na:1.8.0_45] at javax.swing.RepaintManager.access$1300(Unknown Source) [na:1.8.0_45] at javax.swing.RepaintManager$ProcessingRunnable.run(Unknown Source) [na:1.8.0_45] at java.awt.event.InvocationEvent.dispatch(Unknown Source) [na:1.8.0_45] at java.awt.EventQueue.dispatchEventImpl(Unknown Source) [na:1.8.0_45] at java.awt.EventQueue.access$500(Unknown Source) [na:1.8.0_45] at java.awt.EventQueue$3.run(Unknown Source) [na:1.8.0_45] at java.awt.EventQueue$3.run(Unknown Source) [na:1.8.0_45] at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_45] at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source) [na:1.8.0_45] at java.awt.EventQueue.dispatchEvent(Unknown Source) [na:1.8.0_45] at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source) [na:1.8.0_45] at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source) [na:1.8.0_45] at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source) [na:1.8.0_45] at java.awt.EventDispatchThread.pumpEvents(Unknown Source) [na:1.8.0_45] at java.awt.EventDispatchThread.pumpEvents(Unknown Source) [na:1.8.0_45] at java.awt.EventDispatchThread.run(Unknown Source) [na:1.8.0_45] Caused by: java.lang.NullPointerException: null ... 46 common frames omitted
          Hide
          mason Mason Meyer (Inactive) added a comment -

          Caching has been implemented but there are still issues that require fixing. Instead of commenting on this story when problems arise I am going to close this story and create new JIRA stories when problems arise. Doing this will allow the team to easily see the specific issues that need to be addressed.

          Show
          mason Mason Meyer (Inactive) added a comment - Caching has been implemented but there are still issues that require fixing. Instead of commenting on this story when problems arise I am going to close this story and create new JIRA stories when problems arise. Doing this will allow the team to easily see the specific issues that need to be addressed.

            People

            • Assignee:
              mason Mason Meyer (Inactive)
              Reporter:
              dcnorris David Norris (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: