Details
-
Type:
Task
-
Status: Closed (View Workflow)
-
Priority:
Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:None
-
Story Points:1.5
-
Epic Link:
-
Sprint:Winter 1
Description
Situation: The App Store is currently down. When I try to access apps.bioviz.org, I get a "500 Internal Server Error" which states:
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at admin@apps.bioviz.org to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.
Task: Investigate the App Store's server error logs and find out what's causing the server to crash.
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
Reviewing appstore plabooks in https://bitbucket.org/lorainelab/appstore-playbooks/src/master/.
Restarted apache2 and observed this error in apache's error log:
[Tue Jan 07 15:28:00.336325 2025] [wsgi:error] [pid 17604:tid 139875700123392] [remote 16.98.114.149:48507] MySQLdb._exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'appstore.cyaj0olqvfzp.us-east-1.rds.amazonaws.com' (111)")
Checked RDS database in the AWS console (Web interface). Confirmed it is named "appstore." Its status is listed as "storage full." It is a db.t3.micro
instance.
Looking into how I can increase the storage.
Reviewed this documentation: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.ModifyingExisting.html
Followed instructions:
- Selected appstore database and clicked "Modify" button on RDS console
- Observed that the current setting for "Allocated Storage" is 10 GiB
- Noted that the textbox listed current setting for "Allocated Storage" says: "Allocated storage value must be 20 GiB to 6,144 GiB"
- Decided to increase storage to 30 instead of 10 GiB
- Selected "Continue" to move to the next screen
- Observed that two changes are proposed, not just the size increase. These include the size increase (as in above change) and a change to a new certificate - current value is "rds-ca-2019" and new value will be "rds-ca-rsa2048-g1"
- Selected option "Apply immediately"
- Selected "Modify DB instance"
However, when I selected the option "Modify DB instance" I got an error:
"We're sorry, your request to modify DB instance appstore has failed.
The specified database instance is currently in storage-full state. Please allocate more storage by modifying the DB instance."
According to the AWS console for the RDS service, the current certificate is expired or is about to expire and needs to be updated. There doesn't seem to be a way to use the Web interface to increase storage without also updating the certificate. But replacing / updating the certificate might be impossible without first increasing storage. This is what the error message suggests.
Maybe I can use the AWS command line interface (cli) to increase the storage. After that, I can then update the certificate, maybe.
I'm going to try using the aws cli (command line interface) to increase storage size on the database and then attempt to upgrade the certificate.
Installed aws-cli on my Mac with:
brew install awscli
Checked version with:
local aloraine$ aws --version aws-cli/2.22.29 Python/3.12.8 Darwin/23.3.0 source/arm64
Configuring my local environment, following documentation: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
Accessing RDS database using AWS cli:
aws rds describe-db-instances --db-instance-identifier appstore
returns JSON:
{
"DBInstances": [
{
"DBInstanceIdentifier": "appstore",
"DBInstanceClass": "db.t3.micro",
"Engine": "mysql",
"DBInstanceStatus": "storage-full",
"MasterUsername": "admin",
"Endpoint": {
"Address": "appstore.cyaj0olqvfzp.us-east-1.rds.amazonaws.com",
"Port": 3306,
"HostedZoneId": "Z2R2ITUGPM61AM"
},
"AllocatedStorage": 10,
"InstanceCreateTime": "2021-04-15T15:58:10.706000+00:00",
"PreferredBackupWindow": "06:18-06:48",
"BackupRetentionPeriod": 1,
"DBSecurityGroups": [],
"VpcSecurityGroups": [
{
"VpcSecurityGroupId": "sg-092488fea38ffebbf",
"Status": "active"
}
],
"DBParameterGroups": [
{
"DBParameterGroupName": "default.mysql8.0",
"ParameterApplyStatus": "in-sync"
}
],
"AvailabilityZone": "us-east-1d",
"DBSubnetGroup": {
"DBSubnetGroupName": "default",
"DBSubnetGroupDescription": "default",
"VpcId": "vpc-5095f237",
"SubnetGroupStatus": "Complete",
"Subnets": [
{
"SubnetIdentifier": "subnet-ceb135c2",
"SubnetAvailabilityZone": {
"Name": "us-east-1f"
},
"SubnetOutpost": {},
"SubnetStatus": "Active"
},
{
"SubnetIdentifier": "subnet-16b3884e",
"SubnetAvailabilityZone": {
"Name": "us-east-1b"
},
"SubnetOutpost": {},
"SubnetStatus": "Active"
},
{
"SubnetIdentifier": "subnet-8ec6ceb3",
"SubnetAvailabilityZone": {
"Name": "us-east-1e"
},
"SubnetOutpost": {},
"SubnetStatus": "Active"
},
{
"SubnetIdentifier": "subnet-1a5ec57f",
"SubnetAvailabilityZone": {
"Name": "us-east-1c"
},
"SubnetOutpost": {},
"SubnetStatus": "Active"
},
{
"SubnetIdentifier": "subnet-91f2c9bb",
"SubnetAvailabilityZone": {
"Name": "us-east-1d"
},
"SubnetOutpost": {},
"SubnetStatus": "Active"
},
{
"SubnetIdentifier": "subnet-bf2cc7f6",
"SubnetAvailabilityZone": {
"Name": "us-east-1a"
},
"SubnetOutpost": {},
"SubnetStatus": "Active"
}
]
},
"PreferredMaintenanceWindow": "tue:05:01-tue:05:31",
"PendingModifiedValues": {},
"LatestRestorableTime": "2024-12-27T10:15:00+00:00",
"MultiAZ": false,
"EngineVersion": "8.0.35",
"AutoMinorVersionUpgrade": true,
"ReadReplicaDBInstanceIdentifiers": [],
"LicenseModel": "general-public-license",
"OptionGroupMemberships": [
{
"OptionGroupName": "default:mysql-8-0",
"Status": "in-sync"
}
],
"PubliclyAccessible": false,
"StorageType": "gp2",
"DbInstancePort": 0,
"StorageEncrypted": false,
"DbiResourceId": "db-LESFHGQX7IFECVSANMGFA4EMHA",
"CACertificateIdentifier": "rds-ca-2019",
"DomainMemberships": [],
"CopyTagsToSnapshot": false,
"MonitoringInterval": 0,
"DBInstanceArn": "arn:aws:rds:us-east-1:702105950212:db:appstore",
"IAMDatabaseAuthenticationEnabled": false,
"PerformanceInsightsEnabled": false,
"DeletionProtection": false,
"AssociatedRoles": [],
"TagList": [
{
"Key": "Environment",
"Value": "Development"
}
],
"CustomerOwnedIpEnabled": false,
"ActivityStreamStatus": "stopped",
"BackupTarget": "region",
"NetworkType": "IPV4",
"CertificateDetails": {
"CAIdentifier": "rds-ca-2019",
"ValidTill": "2024-08-22T17:08:50+00:00"
},
"DedicatedLogVolume": false,
"IsStorageConfigUpgradeAvailable": true,
"EngineLifecycleSupport": "open-source-rds-extended-support"
}
]
}
To increase size from 10 GiB to 30 GiB, following guide at: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Troubleshooting.html#CHAP_Troubleshooting.Storage
To increase storage without doing anything else, I ran this command:
aws rds modify-db-instance --db-instance-identifier appstore --allocated-storage 30 --apply-immediately > modify.txt
After a few minutes, checked status of the RDS database appstore using the AWS console - the Web site. Observed that the status for RDS instance "appstore" is now listed as "Backing-up" instead of "storage full".
Time is now: 10:59 am CST.
Taking a break.
Time is now: 10:05 am CST.
Status is now "Storage-optimization (0%)"
Turned off apache2 web server to ensure it doesn't attempt to connect to the database during storage modifications currently underway.
Time is now: 11:15 am
Status is now "Available"
Made snapshot: appstore-2025-01-07
Logged back into apps.bioviz.org with:
ssh ubuntu@apps.bioviz.org
and started the apache2 server with:
prodappstore1 ubuntu $ sudo su prodappstore1 root $ service apache2 start
Visited web site https://apps.bioviz.org and observed that the App Store interface appeared as it should.
Next, updating the certificate.
For this, I selected the appstore RDS instance in the RDS Web console.
I selected "Modify" and proceeded to the next screen, which (by default) showed replacing the existing expired certificate with a new one.
Expired certificate current value is: rds-ca-2019
New value is: rds-ca-rsa2048-g1
Selected "Apply immediately" for the "when change will take effect" option.
Stopped apache2 web server.
Selected the modify button and observed the status changed to "modifying."
Status is now: "available."
Restarted apache2 web server.
Observed that https://apps.bioviz.org is on-line.
So far as I know, the site is now back! Thank you Paige Kulzer for alerting me to the problem.
Moving to first level review.
Suggestions for testing:
- Click through the site and make note of anything that looks wrong
- Check that IGB is able to install Apps.
Each app can be clicked on in the App Store and the description for each app loads properly. Signing into the App Store works without issue and all links/buttons appear to be working as expected. Launching the IGB App Manager in IGB 10.1.0 on Mac, all available apps are listed and can be installed without issue. I also checked several apps after installation to ensure that they are functioning as expected.
It looks like the error has been fixed!
Logged into host and discovered debug.log file /home/ubuntu/appstore had grown to 17 Gb.
Deleted the file. 500 error still in effect.
Current host is EC2 with name "prodappstore2" and public IP address 54.84.250.132.
Checked apache2 status after switching to root:
Most recent restart for apache2 was Oct 3, 2024.
Checked disk:
App store code is in /home/ubuntu/appstore.
Contents of /home/ubuntu/appstore total 276M.
Contents of /home/ubuntu total 335M.