Details
-
Type:
Task
-
Status: Closed (View Workflow)
-
Priority:
Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:None
-
Story Points:3
-
Epic Link:
-
Sprint:Fall 4 2022 Oct 10, Fall 5 2022 Oct 24, Spring 6 2023 Mar 20
Description
Situation: Jira keeps going down with a 503.
Task: Determine why Jira is going down.
Note:
To restart the Jira server:
- log into the jira host using ssh as user "ec2-user"
- change to root user (sudo su)
- change to the "bin" directory, located here: /home/ec2-user/jira/atlassian-jira-software-7.0.11-standalone/bin
- let the server shut down properly, just in case it is still running, by executing the "stop jira" script: stop-jira.sh
Note that if the server has crashed, probably there is still a "stale" PID file. If so, when you attempt to stop jira, the script will print that there is a stale PID file.
- Restart jira by running the "start jira" script: start-jira.sh
To enable Nowlan Freese to carry out the above workflow I did this:
- Added his public key to the authorized hosts on jira.bioviz.org
- Modifed the EC2 and its attached security group "jira3" to enable him to restart the the EC2 and also modify its security group
- Used the IAM policy simulator on AWS to check if his user name can edit the security group, which confirmed that he can do it.
Attachments
Activity
I am able to modify the security group for the jira3 EC2 and I am able to ssh onto the server.
Closing ticket.
Having problems again with the jira host. Error message from hs_err_pid2261.log:
- There is insufficient memory for the Java Runtime Environment to continue.
- Native memory allocation (mmap) failed to map 7340032 bytes for committing res
erved memory.- Possible reasons:
- The system is out of physical RAM or swap space
- The process is running with CompressedOops enabled, and the Java Heap may be
blocking the growth of the native heap
Host has little physical memory left:
jira.bioviz.org ec2-user $ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 2.5G 0 2.5G 0% /dev tmpfs 2.5G 0 2.5G 0% /dev/shm tmpfs 2.5G 41M 2.4G 2% /run tmpfs 2.5G 0 2.5G 0% /sys/fs/cgroup /dev/nvme0n1p1 100G 96G 4.3G 96% / tmpfs 497M 0 497M 0% /run/user/1000
Modifying volume. Increasing to 150 Gb up from 100 Gb.
Need to extend Linux file system after resizing according to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-modify-volume.html
Increased disk space and rebooted EC2. New space is available:
jira.bioviz.org ec2-user $ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 2.5G 0 2.5G 0% /dev tmpfs 2.5G 0 2.5G 0% /dev/shm tmpfs 2.5G 400K 2.5G 1% /run tmpfs 2.5G 0 2.5G 0% /sys/fs/cgroup /dev/nvme0n1p1 150G 95G 56G 64% / tmpfs 497M 0 497M 0% /run/user/1000
Checked the the httpd server was running. It was not. Started it and then started jira and confluence manually using vendor-provided startup scripts.
Now checking backup S3 bucket to see if backups stop being made.
They were indeed made. Deleted stale backups.
Jira and confluence (https://wiki.bioviz.org/confluence/display/igbman) are both back up. Moving to closed.
Upon casual inspection, I did not determine why the site crashed. However, I did notice that our support license had expired. I requested a new license for "Jira software" using the Atlassian Web site. The order number is AT-205014171. Clicking on my account page at Atlassian revealed the new license, which I entered into the Jira administration page. I had to make a second order for the "Jira core". That order number was AT-205014270. I obtained the new licenses and added them to the site using an "admin" screen.