Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-3201

Investigate why Jira goes down with a 503

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      Situation: Jira keeps going down with a 503.

      Task: Determine why Jira is going down.

      Note:

      To restart the Jira server:

      • log into the jira host using ssh as user "ec2-user"
      • change to root user (sudo su)
      • change to the "bin" directory, located here: /home/ec2-user/jira/atlassian-jira-software-7.0.11-standalone/bin
      • let the server shut down properly, just in case it is still running, by executing the "stop jira" script: stop-jira.sh

      Note that if the server has crashed, probably there is still a "stale" PID file. If so, when you attempt to stop jira, the script will print that there is a stale PID file.

      • Restart jira by running the "start jira" script: start-jira.sh

      To enable Nowlan Freese to carry out the above workflow I did this:

      • Added his public key to the authorized hosts on jira.bioviz.org
      • Modifed the EC2 and its attached security group "jira3" to enable him to restart the the EC2 and also modify its security group
      • Used the IAM policy simulator on AWS to check if his user name can edit the security group, which confirmed that he can do it.

        Attachments

          Activity

          Hide
          nfreese Nowlan Freese added a comment -

          I am able to modify the security group for the jira3 EC2 and I am able to ssh onto the server.

          Closing ticket.

          Show
          nfreese Nowlan Freese added a comment - I am able to modify the security group for the jira3 EC2 and I am able to ssh onto the server. Closing ticket.
          Hide
          ann.loraine Ann Loraine added a comment - - edited

          Having problems again with the jira host. Error message from hs_err_pid2261.log:

          1. There is insufficient memory for the Java Runtime Environment to continue.
          2. Native memory allocation (mmap) failed to map 7340032 bytes for committing res
            erved memory.
          3. Possible reasons:
          4. The system is out of physical RAM or swap space
          5. The process is running with CompressedOops enabled, and the Java Heap may be
            blocking the growth of the native heap

          Host has little physical memory left:

          jira.bioviz.org ec2-user $ df -h
          Filesystem      Size  Used Avail Use% Mounted on
          devtmpfs        2.5G     0  2.5G   0% /dev
          tmpfs           2.5G     0  2.5G   0% /dev/shm
          tmpfs           2.5G   41M  2.4G   2% /run
          tmpfs           2.5G     0  2.5G   0% /sys/fs/cgroup
          /dev/nvme0n1p1  100G   96G  4.3G  96% /
          tmpfs           497M     0  497M   0% /run/user/1000
          
          Show
          ann.loraine Ann Loraine added a comment - - edited Having problems again with the jira host. Error message from hs_err_pid2261.log: There is insufficient memory for the Java Runtime Environment to continue. Native memory allocation (mmap) failed to map 7340032 bytes for committing res erved memory. Possible reasons: The system is out of physical RAM or swap space The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap Host has little physical memory left: jira.bioviz.org ec2-user $ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 2.5G 0 2.5G 0% /dev tmpfs 2.5G 0 2.5G 0% /dev/shm tmpfs 2.5G 41M 2.4G 2% /run tmpfs 2.5G 0 2.5G 0% /sys/fs/cgroup /dev/nvme0n1p1 100G 96G 4.3G 96% / tmpfs 497M 0 497M 0% /run/user/1000
          Hide
          ann.loraine Ann Loraine added a comment -

          Modifying volume. Increasing to 150 Gb up from 100 Gb.
          Need to extend Linux file system after resizing according to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-modify-volume.html

          Show
          ann.loraine Ann Loraine added a comment - Modifying volume. Increasing to 150 Gb up from 100 Gb. Need to extend Linux file system after resizing according to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-modify-volume.html
          Hide
          ann.loraine Ann Loraine added a comment -

          Increased disk space and rebooted EC2. New space is available:

          jira.bioviz.org ec2-user $ df -h
          Filesystem      Size  Used Avail Use% Mounted on
          devtmpfs        2.5G     0  2.5G   0% /dev
          tmpfs           2.5G     0  2.5G   0% /dev/shm
          tmpfs           2.5G  400K  2.5G   1% /run
          tmpfs           2.5G     0  2.5G   0% /sys/fs/cgroup
          /dev/nvme0n1p1  150G   95G   56G  64% /
          tmpfs           497M     0  497M   0% /run/user/1000
          

          Checked the the httpd server was running. It was not. Started it and then started jira and confluence manually using vendor-provided startup scripts.

          Now checking backup S3 bucket to see if backups stop being made.

          They were indeed made. Deleted stale backups.

          Show
          ann.loraine Ann Loraine added a comment - Increased disk space and rebooted EC2. New space is available: jira.bioviz.org ec2-user $ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 2.5G 0 2.5G 0% /dev tmpfs 2.5G 0 2.5G 0% /dev/shm tmpfs 2.5G 400K 2.5G 1% /run tmpfs 2.5G 0 2.5G 0% /sys/fs/cgroup /dev/nvme0n1p1 150G 95G 56G 64% / tmpfs 497M 0 497M 0% /run/user/1000 Checked the the httpd server was running. It was not. Started it and then started jira and confluence manually using vendor-provided startup scripts. Now checking backup S3 bucket to see if backups stop being made. They were indeed made. Deleted stale backups.
          Hide
          ann.loraine Ann Loraine added a comment -

          Jira and confluence (https://wiki.bioviz.org/confluence/display/igbman) are both back up. Moving to closed.

          Show
          ann.loraine Ann Loraine added a comment - Jira and confluence ( https://wiki.bioviz.org/confluence/display/igbman ) are both back up. Moving to closed.

            People

            • Assignee:
              ann.loraine Ann Loraine
              Reporter:
              nfreese Nowlan Freese
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: