Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-3076

Get help and information from CyVerse regarding why the new app won't properly run

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Story Points:
      0.5
    • Sprint:
      Spring 3 2022 Jan 31 - Feb 11, Spring 4 2022 Feb 14 - Feb 25, Spring 5 2022 Feb 28 - Mar 11, Spring 6 2022 Mar 14 - Mar 25, Spring 7 2022 Mar 28 - Apr 8

      Description

      Karthik Raveendran has created a new Docker image containing deepTools, tabix (samtools), and a custom script he wrote to enable running of the image in CyVerse environment to compute a scaled coverage graph file (bedgraph format) and then tabix-index it.

      He's run into some problems with getting the new image to run as wanted in the environment.

      He will send a email asking about this to the CyVerse team and record the answers and insights obtained here.

      Questions:

      • How do I get access to the condor stderr and stdout ?
      • How do I add metadata tags to my app even while it's in my private account and not yet widely available, i.e., "published"

      This may be helpful:

      IGB class information about using the IGB Docker image to build IGB:
      https://canvas.instructure.com/courses/1164217/pages/learn-to-use-igb-docker-image-build-environment?module_item_id=43241015

        Attachments

          Issue Links

            Activity

            Hide
            karthik Karthik Raveendran added a comment -

            Cyverse support team was contacted yesterday via the chat module regarding the questions in the description. They did not reply so a mail was sent to support@cyverse.org. Meanwhile, in the Cyverse app, bam2tbi, I was able to send file path to the input parameter but the image stopped running. I was not able to discern the nature to the error.

            Show
            karthik Karthik Raveendran added a comment - Cyverse support team was contacted yesterday via the chat module regarding the questions in the description. They did not reply so a mail was sent to support@cyverse.org. Meanwhile, in the Cyverse app, bam2tbi, I was able to send file path to the input parameter but the image stopped running. I was not able to discern the nature to the error.
            Hide
            karthik Karthik Raveendran added a comment -

            Amanda from Cyverse Support:
            "Apps (non-Agave) can be added to collections. These function much like tags by grouping the apps together for easy findability. To create/manage collections go to the 'collections' tab on the left side of the Discovery Environment (the icon looks like a little building)."

            There is a Integrated Genome Browser collection managed my Nowlan Freese where the tool can be added

            Show
            karthik Karthik Raveendran added a comment - Amanda from Cyverse Support: "Apps (non-Agave) can be added to collections. These function much like tags by grouping the apps together for easy findability. To create/manage collections go to the 'collections' tab on the left side of the Discovery Environment (the icon looks like a little building)." There is a Integrated Genome Browser collection managed my Nowlan Freese where the tool can be added
            Hide
            karthik Karthik Raveendran added a comment -

            Cyverse Support Team:
            The condor stderr and stdout files should be included in the logs folder once the analysis completes (or fails). They will have odd file names like "condor-stdout-0" and "condor-stderr-0" (without the quotes).
            If the condor-stderr-0 and condor-stdout-0 files are missing, then there might be something wrong that's causing the job to fail before the tool actually executes. Some common issues we've seen are input files being moved in the data store before the analysis gets a chance to execute or the Docker image repo being incorrect in the tool definition. You may find more info along those lines in the docker-compose related log files in the logs folder.
            If you still can't find the problem, I'd recommend sharing the analysis with support by clicking on the analysis status on the Analyses page and following the instructions that pop up.

            Show
            karthik Karthik Raveendran added a comment - Cyverse Support Team: The condor stderr and stdout files should be included in the logs folder once the analysis completes (or fails). They will have odd file names like "condor-stdout-0" and "condor-stderr-0" (without the quotes). If the condor-stderr-0 and condor-stdout-0 files are missing, then there might be something wrong that's causing the job to fail before the tool actually executes. Some common issues we've seen are input files being moved in the data store before the analysis gets a chance to execute or the Docker image repo being incorrect in the tool definition. You may find more info along those lines in the docker-compose related log files in the logs folder. If you still can't find the problem, I'd recommend sharing the analysis with support by clicking on the analysis status on the Analyses page and following the instructions that pop up.
            Hide
            karthik Karthik Raveendran added a comment -

            Cyverse Support Team:
            If you share your analysis with 'siuser' (see instructions below) I can have a look at it. If it's something I recognize I will certainly let you know. If it is something specific to your app/tool I may not be much help.

            You can share your analysis by opening the Analyses window and clicking on the corresponding Status for your analysis. This will take you through several troubleshooting steps. If the troubleshooting steps do not solve your problem click the button that says "I still need help!". Clicking this button will present a form you'll need to fill out in order to share your analysis with us.

            Show
            karthik Karthik Raveendran added a comment - Cyverse Support Team: If you share your analysis with 'siuser' (see instructions below) I can have a look at it. If it's something I recognize I will certainly let you know. If it is something specific to your app/tool I may not be much help. You can share your analysis by opening the Analyses window and clicking on the corresponding Status for your analysis. This will take you through several troubleshooting steps. If the troubleshooting steps do not solve your problem click the button that says "I still need help!". Clicking this button will present a form you'll need to fill out in order to share your analysis with us.
            Hide
            karthik Karthik Raveendran added a comment -

            The docker image pushed to docker hub repository needs modifications because the documentation for developing the Cyverse app does not specify how to use an interactive docker image connecting the app UI and run shell scripts within it. The example mentioned in the documentation uses an executable docker image although "interactive" is an option. For now, the image can be modified to specify the .tbi generator script as an "entrypoint" so that the image is executable.

            Show
            karthik Karthik Raveendran added a comment - The docker image pushed to docker hub repository needs modifications because the documentation for developing the Cyverse app does not specify how to use an interactive docker image connecting the app UI and run shell scripts within it. The example mentioned in the documentation uses an executable docker image although "interactive" is an option. For now, the image can be modified to specify the .tbi generator script as an "entrypoint" so that the image is executable.
            Hide
            karthik Karthik Raveendran added a comment - - edited

            The Cyverse app failed several times although the input file argument is being correctly passed. However, there is no obvious option in Cyverse app development to mount the local directory which is required for the script to run. So the new strategy is that the igb_tbi_creator.sh is given as the ENTRYPOINT of the container and the argument is overridden as shown here .
            However, I believe for the Cyverse app to run, the command that Cyverse would use is a simple "docker run <container_name> <input_argument_array>". Therefore overriding the ENTRYPOINT is difficult without fully understanding the internal workings of Cyverse app development. So a combination of ENTRYPOINT and CMD(for the arguments) is used and it works in its simplest form in the local system and needs testing in the Cyverse environment. The goal is to keep the docker run command as simple as possible as it cannot be used to dynamically change things in the way we run the image. Another assumption is that the Cyverse DE Data environment is automatically mounted when the app is run. If not the input folder can be specified.
            Additionally, a numpy error was also found in the bamCoverage so that was addressed as well.

            Show
            karthik Karthik Raveendran added a comment - - edited The Cyverse app failed several times although the input file argument is being correctly passed. However, there is no obvious option in Cyverse app development to mount the local directory which is required for the script to run. So the new strategy is that the igb_tbi_creator.sh is given as the ENTRYPOINT of the container and the argument is overridden as shown here . However, I believe for the Cyverse app to run, the command that Cyverse would use is a simple "docker run <container_name> <input_argument_array>". Therefore overriding the ENTRYPOINT is difficult without fully understanding the internal workings of Cyverse app development. So a combination of ENTRYPOINT and CMD(for the arguments) is used and it works in its simplest form in the local system and needs testing in the Cyverse environment. The goal is to keep the docker run command as simple as possible as it cannot be used to dynamically change things in the way we run the image. Another assumption is that the Cyverse DE Data environment is automatically mounted when the app is run. If not the input folder can be specified. Additionally, a numpy error was also found in the bamCoverage so that was addressed as well.
            Hide
            nfreese Nowlan Freese added a comment - - edited

            [~aloraine] - It may be worth contacting Sriram Srinivasan and seeing if he would be willing to meet up or have a zoom call. He is located in Cary, NC, so he is not far from us.

            Software Engineer Lead, Core Software, University of Arizona | sriram@cyverse.org | https://cyverse.org/team
            Linkedin profile: https://www.linkedin.com/in/sriramsrinivasan83/

            Show
            nfreese Nowlan Freese added a comment - - edited [~aloraine] - It may be worth contacting Sriram Srinivasan and seeing if he would be willing to meet up or have a zoom call. He is located in Cary, NC, so he is not far from us. Software Engineer Lead, Core Software, University of Arizona | sriram@cyverse.org | https://cyverse.org/team Linkedin profile: https://www.linkedin.com/in/sriramsrinivasan83/
            Hide
            karthik Karthik Raveendran added a comment -

            I sent and email to Sriram in Cyverse with an image attachment (added here as well):
            "I am part of the BioViz Connect Team from the Genome Visualization Lab in Charlotte. I was trying to deploy an app in Cyverse using the tool bam2tbi and the job keeps failing. I checked the logs for ‘condor_stderr’ file and ‘condor_stdout’ file but it was not there. I made changes to the docker file, assuming what could be the errors and it continued to fail. I was really hoping to get some help with this. Thank you in advance.

            Moreover, I was wondering how to add the ‘Argument’ parameter in the Cyverse app if its possible. I have attached an image of an app that the Cyverse team had created for us which illustrates what I mean by the ‘Argument’ parameter

            Regards,
            Karthik Raveendran"

            Show
            karthik Karthik Raveendran added a comment - I sent and email to Sriram in Cyverse with an image attachment (added here as well): "I am part of the BioViz Connect Team from the Genome Visualization Lab in Charlotte. I was trying to deploy an app in Cyverse using the tool bam2tbi and the job keeps failing. I checked the logs for ‘condor_stderr’ file and ‘condor_stdout’ file but it was not there. I made changes to the docker file, assuming what could be the errors and it continued to fail. I was really hoping to get some help with this. Thank you in advance. Moreover, I was wondering how to add the ‘Argument’ parameter in the Cyverse app if its possible. I have attached an image of an app that the Cyverse team had created for us which illustrates what I mean by the ‘Argument’ parameter Regards, Karthik Raveendran"
            Hide
            ann.loraine Ann Loraine added a comment -

            Any answer Karthik Raveendran?

            Show
            ann.loraine Ann Loraine added a comment - Any answer Karthik Raveendran ?
            Hide
            karthik Karthik Raveendran added a comment -

            I have not received a reply for the above.

            Show
            karthik Karthik Raveendran added a comment - I have not received a reply for the above.
            Hide
            ann.loraine Ann Loraine added a comment -

            As per scrum discussion today, closing this ticket as we are not getting a reply from CyVerse support. We will revisit when we have greater need to create new CyVerse apps.

            Also, let's be sure to attend any new tutorials at conferences on how to create apps. (comment from Karthik Raveendran)

            Show
            ann.loraine Ann Loraine added a comment - As per scrum discussion today, closing this ticket as we are not getting a reply from CyVerse support. We will revisit when we have greater need to create new CyVerse apps. Also, let's be sure to attend any new tutorials at conferences on how to create apps. (comment from Karthik Raveendran )

              People

              • Assignee:
                karthik Karthik Raveendran
                Reporter:
                ann.loraine Ann Loraine
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: