[IGBF-4043] Fix App Store Internal Server Error - JIRA UNCC

Details

Type: Task
Status: Closed (View Workflow)
Priority: Major
Resolution: Done
Affects Version/s: None
Fix Version/s: None
Labels:
None

Story Points:
1.5
Epic Link:
Deploy and maintain infrastructure
Sprint:
Winter 1

Description

Situation: The App Store is currently down. When I try to access apps.bioviz.org, I get a "500 Internal Server Error" which states:

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at admin@apps.bioviz.org to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Task: Investigate the App Store's server error logs and find out what's causing the server to crash.

Attachments

Issue Links

relates to

IGBF-4041 Update the public about the IGB App Store being down

Closed

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Ann Loraine added a comment - 07/Jan/25 10:14 AM - edited

Logged into host and discovered debug.log file /home/ubuntu/appstore had grown to 17 Gb.

Deleted the file. 500 error still in effect.

Current host is EC2 with name "prodappstore2" and public IP address 54.84.250.132.

Checked apache2 status after switching to root:

system apache2 status

Most recent restart for apache2 was Oct 3, 2024.

Checked disk:

prodappstore1 root $ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            473M     0  473M   0% /dev
tmpfs            98M   11M   87M  11% /run
/dev/xvda1       24G   24G     0 100% /
tmpfs           488M     0  488M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           488M     0  488M   0% /sys/fs/cgroup
/dev/loop1      104M  104M     0 100% /snap/core/16928
/dev/loop3       26M   26M     0 100% /snap/amazon-ssm-agent/7993
/dev/loop4      105M  105M     0 100% /snap/core/17200
/dev/loop2       56M   56M     0 100% /snap/core18/2829
/dev/loop6       56M   56M     0 100% /snap/core18/2846
/dev/loop5       75M   75M     0 100% /snap/core22/1621
/dev/loop7       26M   26M     0 100% /snap/amazon-ssm-agent/9565
tmpfs            98M     0   98M   0% /run/user/1000

App store code is in /home/ubuntu/appstore.
Contents of /home/ubuntu/appstore total 276M.
Contents of /home/ubuntu total 335M.

Show

Ann Loraine added a comment - 07/Jan/25 10:14 AM - edited Logged into host and discovered debug.log file /home/ubuntu/appstore had grown to 17 Gb. Deleted the file. 500 error still in effect. Current host is EC2 with name "prodappstore2" and public IP address 54.84.250.132. Checked apache2 status after switching to root: system apache2 status Most recent restart for apache2 was Oct 3, 2024. Checked disk: prodappstore1 root $ df -h Filesystem Size Used Avail Use% Mounted on udev 473M 0 473M 0% /dev tmpfs 98M 11M 87M 11% /run /dev/xvda1 24G 24G 0 100% / tmpfs 488M 0 488M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 488M 0 488M 0% /sys/fs/cgroup /dev/loop1 104M 104M 0 100% /snap/core/16928 /dev/loop3 26M 26M 0 100% /snap/amazon-ssm-agent/7993 /dev/loop4 105M 105M 0 100% /snap/core/17200 /dev/loop2 56M 56M 0 100% /snap/core18/2829 /dev/loop6 56M 56M 0 100% /snap/core18/2846 /dev/loop5 75M 75M 0 100% /snap/core22/1621 /dev/loop7 26M 26M 0 100% /snap/amazon-ssm-agent/9565 tmpfs 98M 0 98M 0% /run/user/1000 App store code is in /home/ubuntu/appstore. Contents of /home/ubuntu/appstore total 276M. Contents of /home/ubuntu total 335M.

11 older comments

Hide

Permalink

Ann Loraine added a comment - 08/Jan/25 12:14 AM

Made snapshot: appstore-2025-01-07

Show

Ann Loraine added a comment - 08/Jan/25 12:14 AM Made snapshot: appstore-2025-01-07

Hide

Permalink

Ann Loraine added a comment - 08/Jan/25 12:20 AM

Logged back into apps.bioviz.org with:

ssh ubuntu@apps.bioviz.org

and started the apache2 server with:

prodappstore1 ubuntu $ sudo su
prodappstore1 root $ service apache2 start

Visited web site https://apps.bioviz.org and observed that the App Store interface appeared as it should.

Show

Ann Loraine added a comment - 08/Jan/25 12:20 AM Logged back into apps.bioviz.org with: ssh ubuntu@apps.bioviz.org and started the apache2 server with: prodappstore1 ubuntu $ sudo su prodappstore1 root $ service apache2 start Visited web site https://apps.bioviz.org and observed that the App Store interface appeared as it should.

Hide

Permalink

Ann Loraine added a comment - 08/Jan/25 12:20 AM - edited

Next, updating the certificate.
For this, I selected the appstore RDS instance in the RDS Web console.
I selected "Modify" and proceeded to the next screen, which (by default) showed replacing the existing expired certificate with a new one.

Expired certificate current value is: rds-ca-2019
New value is: rds-ca-rsa2048-g1
Selected "Apply immediately" for the "when change will take effect" option.
Stopped apache2 web server.
Selected the modify button and observed the status changed to "modifying."
Status is now: "available."
Restarted apache2 web server.
Observed that https://apps.bioviz.org is on-line.

Show

Ann Loraine added a comment - 08/Jan/25 12:20 AM - edited Next, updating the certificate. For this, I selected the appstore RDS instance in the RDS Web console. I selected "Modify" and proceeded to the next screen, which (by default) showed replacing the existing expired certificate with a new one. Expired certificate current value is: rds-ca-2019 New value is: rds-ca-rsa2048-g1 Selected "Apply immediately" for the "when change will take effect" option. Stopped apache2 web server. Selected the modify button and observed the status changed to "modifying." Status is now: "available." Restarted apache2 web server. Observed that https://apps.bioviz.org is on-line.

Hide

Permalink

Ann Loraine added a comment - 08/Jan/25 12:29 AM

So far as I know, the site is now back! Thank you Paige Kulzer for alerting me to the problem.
Moving to first level review.

Suggestions for testing:

Click through the site and make note of anything that looks wrong
Check that IGB is able to install Apps.

Show

Ann Loraine added a comment - 08/Jan/25 12:29 AM So far as I know, the site is now back! Thank you Paige Kulzer for alerting me to the problem. Moving to first level review. Suggestions for testing: Click through the site and make note of anything that looks wrong Check that IGB is able to install Apps.

Hide

Permalink

Paige Kulzer added a comment - 08/Jan/25 8:41 AM

Each app can be clicked on in the App Store and the description for each app loads properly. Signing into the App Store works without issue and all links/buttons appear to be working as expected. Launching the IGB App Manager in IGB 10.1.0 on Mac, all available apps are listed and can be installed without issue. I also checked several apps after installation to ensure that they are functioning as expected.

It looks like the error has been fixed!

Show

Paige Kulzer added a comment - 08/Jan/25 8:41 AM Each app can be clicked on in the App Store and the description for each app loads properly. Signing into the App Store works without issue and all links/buttons appear to be working as expected. Launching the IGB App Manager in IGB 10.1.0 on Mac, all available apps are listed and can be installed without issue. I also checked several apps after installation to ensure that they are functioning as expected. It looks like the error has been fixed!

People

Assignee:

Ann Loraine

Reporter:

Paige Kulzer

Votes:

0 Vote for this issue

Watchers:

2 Start watching this issue

Dates

Created:

07/Jan/25 10:04 AM

Updated:

08/Jan/25 10:45 AM

Resolved:

08/Jan/25 10:45 AM