[IGBF-3201] Investigate why Jira goes down with a 503 - JIRA UNCC

Nowlan Freese created issue - 17/Oct/22 10:42 AM

Nowlan Freese made changes - 17/Oct/22 10:42 AM

Field	Original Value	New Value
Epic Link		IGBF-2323 [ 18477 ]

Ann Loraine made changes - 19/Oct/22 4:12 PM

Status

To-Do [ 10305 ]

In Progress [ 3 ]

Hide

Permalink

Ann Loraine added a comment - 19/Oct/22 4:24 PM - edited

Upon casual inspection, I did not determine why the site crashed. However, I did notice that our support license had expired. I requested a new license for "Jira software" using the Atlassian Web site. The order number is AT-205014171. Clicking on my account page at Atlassian revealed the new license, which I entered into the Jira administration page. I had to make a second order for the "Jira core". That order number was AT-205014270. I obtained the new licenses and added them to the site using an "admin" screen.

Show

Ann Loraine added a comment - 19/Oct/22 4:24 PM - edited Upon casual inspection, I did not determine why the site crashed. However, I did notice that our support license had expired. I requested a new license for "Jira software" using the Atlassian Web site. The order number is AT-205014171. Clicking on my account page at Atlassian revealed the new license, which I entered into the Jira administration page. I had to make a second order for the "Jira core". That order number was AT-205014270. I obtained the new licenses and added them to the site using an "admin" screen.

Hide

Permalink

Ann Loraine added a comment - 19/Oct/22 4:29 PM

Licenses are now up-to-date until Oct 2023.

Show

Ann Loraine added a comment - 19/Oct/22 4:29 PM Licenses are now up-to-date until Oct 2023.

Hide

Permalink

Ann Loraine added a comment - 19/Oct/22 4:32 PM

Found this page with common causes for jira tomcat crashing:

https://confluence.atlassian.com/jirakb/common-causes-for-jira-server-crashes-and-performance-issues-203394749.html

When I logged onto the server after it crashed, I observed there was a "stale" process id for tomcat. Restarting the server was easy using the provided script, however. The presence of the stale PID file did not affect my attempt to restart the Jira/tomcat process.

Show

Ann Loraine added a comment - 19/Oct/22 4:32 PM Found this page with common causes for jira tomcat crashing: https://confluence.atlassian.com/jirakb/common-causes-for-jira-server-crashes-and-performance-issues-203394749.html When I logged onto the server after it crashed, I observed there was a "stale" process id for tomcat. Restarting the server was easy using the provided script, however. The presence of the stale PID file did not affect my attempt to restart the Jira/tomcat process.

Hide

Permalink

Ann Loraine added a comment - 20/Oct/22 5:35 AM - edited

Page on "java crashes" recommends looking for java crash logs in the "bin" directory of Jira software.
On our current system setup, this "bin" directory is here:

/home/ec2-user/jira/atlassian-jira-software-7.0.11-standalone/bin

There are three such java crash log files:

/home/ec2-user/jira/atlassian-jira-software-7.0.11-standalone/bin
jira.bioviz.org ec2-user $ ls -lh *.log
-rw-r--r-- 1 root root 531K Oct 16 03:58 hs_err_pid23961.log
-rw-r--r-- 1 root root  20K Feb  2  2022 hs_err_pid28078.log
-rw-r--r-- 1 root root 544K Oct 14 01:05 hs_err_pid2960.log

The top part of each file:

jira.bioviz.org ec2-user $ head *.log
==> hs_err_pid23961.log <==
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full

==> hs_err_pid28078.log <==
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 268435456 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full

==> hs_err_pid2960.log <==
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 24641536 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full

Show

Ann Loraine added a comment - 20/Oct/22 5:35 AM - edited Page on "java crashes" recommends looking for java crash logs in the "bin" directory of Jira software. On our current system setup, this "bin" directory is here: /home/ec2-user/jira/atlassian-jira-software-7.0.11-standalone/bin There are three such java crash log files: /home/ec2-user/jira/atlassian-jira-software-7.0.11-standalone/bin jira.bioviz.org ec2-user $ ls -lh *.log -rw-r--r-- 1 root root 531K Oct 16 03:58 hs_err_pid23961.log -rw-r--r-- 1 root root 20K Feb 2 2022 hs_err_pid28078.log -rw-r--r-- 1 root root 544K Oct 14 01:05 hs_err_pid2960.log The top part of each file: jira.bioviz.org ec2-user $ head *.log ==> hs_err_pid23961.log <== # # There is insufficient memory for the Java Runtime Environment to continue . # Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full ==> hs_err_pid28078.log <== # # There is insufficient memory for the Java Runtime Environment to continue . # Native memory allocation (mmap) failed to map 268435456 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full ==> hs_err_pid2960.log <== # # There is insufficient memory for the Java Runtime Environment to continue . # Native memory allocation (mmap) failed to map 24641536 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full

Hide

Permalink

Ann Loraine added a comment - 20/Oct/22 5:41 AM

The two crashes last week happened because java ran out of memory.

Show

Ann Loraine added a comment - 20/Oct/22 5:41 AM The two crashes last week happened because java ran out of memory.

Ann Loraine made changes - 20/Oct/22 5:46 AM

Description

Situation: Jira keeps going down with a 503.

Task: Determine why Jira is going down.

Situation: Jira keeps going down with a 503.

Task: Determine why Jira is going down.

Note:

To restart the Jira server:

* log into the jira host using ssh as user "ec2-user"
* change to root user (sudo su)
* change to the "bin" directory, located here: /home/ec2-user/jira/atlassian-jira-software-7.0.11-standalone/bin
* let the server shut down properly, just in case it is still running, by executing the "stop jira" script: stop-jira.sh

Note that if the server has crashed, probably there is still a "stale" PID file. If so, when you attempt to stop jira, the script will print that there is a stale PID file.

* Restart jira by running the "start jira" script: start-jira.sh

Ann Loraine made changes - 20/Oct/22 5:46 AM

Status

In Progress [ 3 ]

Needs 1st Level Review [ 10005 ]

Ann Loraine made changes - 20/Oct/22 5:46 AM

Assignee

Ann Loraine [ aloraine ]

Hide

Permalink

Ann Loraine added a comment - 21/Oct/22 10:21 AM - edited

[~aloraine] to add NF public key to host. Also, enable NF user to modify security group for the host.

Show

Ann Loraine added a comment - 21/Oct/22 10:21 AM - edited [~aloraine] to add NF public key to host. Also, enable NF user to modify security group for the host.

Ann Loraine made changes - 21/Oct/22 10:21 AM

Assignee

Ann Loraine [ aloraine ]

Ann Loraine made changes - 21/Oct/22 10:21 AM

Status

Needs 1st Level Review [ 10005 ]

First Level Review in Progress [ 10301 ]

Ann Loraine made changes - 21/Oct/22 10:21 AM

Status

First Level Review in Progress [ 10301 ]

To-Do [ 10305 ]

Ann Loraine made changes - 21/Oct/22 12:28 PM

Status

To-Do [ 10305 ]

In Progress [ 3 ]

Ann Loraine made changes - 21/Oct/22 1:59 PM

Description

Situation: Jira keeps going down with a 503.

Task: Determine why Jira is going down.

Note:

To restart the Jira server:

* log into the jira host using ssh as user "ec2-user"
* change to root user (sudo su)
* change to the "bin" directory, located here: /home/ec2-user/jira/atlassian-jira-software-7.0.11-standalone/bin
* let the server shut down properly, just in case it is still running, by executing the "stop jira" script: stop-jira.sh

Note that if the server has crashed, probably there is still a "stale" PID file. If so, when you attempt to stop jira, the script will print that there is a stale PID file.

* Restart jira by running the "start jira" script: start-jira.sh

Situation: Jira keeps going down with a 503.

Task: Determine why Jira is going down.

Note:

To restart the Jira server:

* log into the jira host using ssh as user "ec2-user"
* change to root user (sudo su)
* change to the "bin" directory, located here: /home/ec2-user/jira/atlassian-jira-software-7.0.11-standalone/bin
* let the server shut down properly, just in case it is still running, by executing the "stop jira" script: stop-jira.sh

Note that if the server has crashed, probably there is still a "stale" PID file. If so, when you attempt to stop jira, the script will print that there is a stale PID file.

* Restart jira by running the "start jira" script: start-jira.sh

To enable [~nfreese] to carry out the above workflow I did this:

* Added his public key to the authorized hosts on jira.bioviz.org
* Modifed the EC2 and its attached security group "jira3" to enable him to restart the the EC2 and also modify its security group
* Used the IAM policy simulator on AWS to check if his user name can edit the security group, which confirmed that he can do it.

Ann Loraine made changes - 21/Oct/22 2:00 PM

Status

In Progress [ 3 ]

Needs 1st Level Review [ 10005 ]

Ann Loraine made changes - 21/Oct/22 2:00 PM

Assignee

Ann Loraine [ aloraine ]

Nowlan Freese made changes - 24/Oct/22 9:39 AM

Assignee

Nowlan Freese [ nfreese ]

Nowlan Freese made changes - 24/Oct/22 9:40 AM

Sprint

Fall 4 2022 Oct 10 [ 156 ]

Fall 4 2022 Oct 10, Fall 5 2022 Oct 24 [ 156, 157 ]

Nowlan Freese made changes - 24/Oct/22 9:40 AM

Rank

Ranked higher

Hide

Permalink

Nowlan Freese added a comment - 26/Oct/22 11:32 AM

I am able to modify the security group for the jira3 EC2 and I am able to ssh onto the server.

Closing ticket.

Show

Nowlan Freese added a comment - 26/Oct/22 11:32 AM I am able to modify the security group for the jira3 EC2 and I am able to ssh onto the server. Closing ticket.