Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-4286

Investigate: Use sshfs to open files on the cluster (or other remote system) in IGB

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      This software, in theory, would let us open files stored on the UNC Charlotte cluster in IGB without first having to copy them over to our personal computer's local file system:

      https://github.com/libfuse/sshfs

      This software, called "sshfs" for "secure shell file system", can "mount" a remote file system that is accessible via ssh, onto your local computer. Once that is done, IGB should then be able to open the remote files as if they were actually local.

      For this task, work with Research Computing and other experts as needed to open BAM files in IGB running on your local computer.

      Report on hurdles you encounter, solutions, issues, etc.

      Links:

        Attachments

          Activity

          Hide
          pkulzer Paige Kulzer added a comment -

          I've followed the instructions at the link in the Description to install SSHFS and its dependent package(s). I've also followed the instructions there to mount a directory on the cluster to my local computer:

          sshfs pkulzer@hpc9.charlotte.edu:/projects/tomato_genome/fnb/nfcore_scrnaseq/results/star/ hpcc_nf/ -ovolname=hpcc_nf -o allow_other,defer_permissions,follow_symlinks,reconnect -ocache=no -onolocalcaches
          

          The files are not appearing even though I'm seeing the remote volume on my computer now, so I will contact the HPC team to troubleshoot from here.

          Show
          pkulzer Paige Kulzer added a comment - I've followed the instructions at the link in the Description to install SSHFS and its dependent package(s). I've also followed the instructions there to mount a directory on the cluster to my local computer: sshfs pkulzer@hpc9.charlotte.edu:/projects/tomato_genome/fnb/nfcore_scrnaseq/results/star/ hpcc_nf/ -ovolname=hpcc_nf -o allow_other,defer_permissions,follow_symlinks,reconnect -ocache=no -onolocalcaches The files are not appearing even though I'm seeing the remote volume on my computer now, so I will contact the HPC team to troubleshoot from here.
          Hide
          pkulzer Paige Kulzer added a comment -

          Help ticket has been submitted:

          Hello! I hope your day's going well.

          I recently ran a pipeline on some genomics data I have on the cluster and the resulting files are quite large (>100 GB in total). Rather than downloading all of this data to my local computer for further analyses, I was hoping to use SSHFS to mount the output folder from the cluster to my local machine. I followed the instructions at this link to install SSHFS and run it: https://docs.icer.msu.edu/Mapping_HPC_drives_with_SSHFS/. I'm not seeing any error messages when I run SSHFS and I do see that my destination folder turns into a Remote Volume as expected, but the files are only displaying in the file finder very briefly before disappearing, after which I'm not able to see them in the file finder or via command line.

          I would love some help troubleshooting why this might not be working as expected. I've tried unmounting the folder, deleting and recreating the folder, then mounting it again but that hasn't worked.

          Cheers,
          Paige

          Show
          pkulzer Paige Kulzer added a comment - Help ticket has been submitted: Hello! I hope your day's going well. I recently ran a pipeline on some genomics data I have on the cluster and the resulting files are quite large (>100 GB in total). Rather than downloading all of this data to my local computer for further analyses, I was hoping to use SSHFS to mount the output folder from the cluster to my local machine. I followed the instructions at this link to install SSHFS and run it: https://docs.icer.msu.edu/Mapping_HPC_drives_with_SSHFS/ . I'm not seeing any error messages when I run SSHFS and I do see that my destination folder turns into a Remote Volume as expected, but the files are only displaying in the file finder very briefly before disappearing, after which I'm not able to see them in the file finder or via command line. I would love some help troubleshooting why this might not be working as expected. I've tried unmounting the folder, deleting and recreating the folder, then mounting it again but that hasn't worked. Cheers, Paige
          Hide
          pkulzer Paige Kulzer added a comment - - edited

          I chatted with Steven today about this issue while waiting for the HPC team to get back to me. He suggested that a more long-term solution to opening remote files would be to integrate a Java SSH library into IGB. He wasn't sure which library that might be, but in theory this would allow UNC-Charlotte folk (and potentially anyone working with computing clusters) to view their files in IGB via SSH without having to spend time downloading files to their machines.

          Show
          pkulzer Paige Kulzer added a comment - - edited I chatted with Steven today about this issue while waiting for the HPC team to get back to me. He suggested that a more long-term solution to opening remote files would be to integrate a Java SSH library into IGB. He wasn't sure which library that might be, but in theory this would allow UNC-Charlotte folk (and potentially anyone working with computing clusters) to view their files in IGB via SSH without having to spend time downloading files to their machines.
          Show
          pkulzer Paige Kulzer added a comment - Java SSH library forums/resources: What is an example of using an SSH library in Java, and how can I establish a secure connection with it? The Evolution of Java SSH Libraries SSH Connection with Java Libraries I've seen mentioned the most: Apache MINA SSHD JSch
          Hide
          pkulzer Paige Kulzer added a comment -

          Here's the response from the HPC team - it looks like sshfs won't be a viable option with the research cluster:

          We do not allow mounting of directories either on our system or of our shared storage onto others. One could do the additional processing on our cluster and we are happy to have that discussion.

          Show
          pkulzer Paige Kulzer added a comment - Here's the response from the HPC team - it looks like sshfs won't be a viable option with the research cluster: We do not allow mounting of directories either on our system or of our shared storage onto others. One could do the additional processing on our cluster and we are happy to have that discussion.

            People

            • Assignee:
              pkulzer Paige Kulzer
              Reporter:
              ann.loraine Ann Loraine
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: