[IGBF-1990] Fix S3 permissions issue - JIRA UNCC

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Blocker
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Labels:
None

Story Points:
1
Epic Link:
Develop App Store for IGB
Sprint:
Fall 2019 Sprint 2

Description

When users upload files, IGB AppStore stores them in an S3 bucket. However, when client software - including both IGB and the user's Web browser - tries to retrieve those files, access is denied.

IGB AppStore should ensure that all files uploaded by users are accessible.

Attachments

Issue Links

relates to

IGBF-1991 Simplify jar file name

Closed

Activity

Ascending order - Click to sort in descending order

Ann Loraine created issue - 02/Sep/19 12:31 PM

Ann Loraine made changes - 02/Sep/19 12:31 PM

Field	Original Value	New Value
Epic Link		IGBF-1388 [ 17463 ]

Ann Loraine made changes - 02/Sep/19 12:31 PM

Rank

Ranked higher

Ann Loraine made changes - 02/Sep/19 12:31 PM

Status

Open [ 1 ]

To-Do [ 10305 ]

Ann Loraine made changes - 02/Sep/19 12:31 PM

Status

To-Do [ 10305 ]

In Progress [ 3 ]

Hide

Permalink

Ann Loraine added a comment - 02/Sep/19 12:36 PM

https://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/
https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html

Show

Ann Loraine added a comment - 02/Sep/19 12:36 PM https://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/ https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html

Hide

Permalink

Ann Loraine added a comment - 02/Sep/19 2:56 PM

testdriven.io links/advice appears out of date
Solution appears to be:
1) turn off block public access
2) add bucket permission:

{
"Version": "2012-10-17",
"Statement": [

{ "Sid": "PublicReadGetObject", "Effect": "Allow", "Principal": "*", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::devappstoreN-media-0000/*" ] }

]
}

Documented same in test appstore setup.

Show

Ann Loraine added a comment - 02/Sep/19 2:56 PM testdriven.io links/advice appears out of date Solution appears to be: 1) turn off block public access 2) add bucket permission: { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadGetObject", "Effect": "Allow", "Principal": "*", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::devappstoreN-media-0000/*" ] } ] } Documented same in test appstore setup.

Hide

Permalink

Ann Loraine added a comment - 02/Sep/19 3:02 PM - edited

Configured devappstore S3 buckets 1 through 6 as in previous comment. Tested with DevAppStore5.
To test:

1) Start up your DevAppStore EC2.
2) Check that images are being rendered properly in your browser.
3) Check that you can download an IGB App jar file from the same URL IGB uses.

To determine the URL that IGB uses, check the URI tag generated by the OBR index file endpoint. Pre-pend the EC2 domain in front of it and try to hit that endpoint.

Show

Ann Loraine added a comment - 02/Sep/19 3:02 PM - edited Configured devappstore S3 buckets 1 through 6 as in previous comment. Tested with DevAppStore5. To test: 1) Start up your DevAppStore EC2. 2) Check that images are being rendered properly in your browser. 3) Check that you can download an IGB App jar file from the same URL IGB uses. To determine the URL that IGB uses, check the URI tag generated by the OBR index file endpoint. Pre-pend the EC2 domain in front of it and try to hit that endpoint.

Ann Loraine made changes - 02/Sep/19 3:22 PM

Status

In Progress [ 3 ]

Needs 1st Level Review [ 10005 ]

Ann Loraine made changes - 02/Sep/19 3:22 PM

Assignee

Ann Loraine [ aloraine ]

Ann Loraine made changes - 02/Sep/19 4:02 PM

Link

This issue relates to IGBF-1991 [ IGBF-1991 ]

Ann Loraine made changes - 03/Sep/19 1:35 PM

Comment

[ When I first tried to test accessing the jar file (step 3 in previous comment), I realized that the actual path to the file in S3 was inconsistent with the OBR index file.

For dev-appstore-5, the OBR index file was:
* https://dev-appstore-5.bioviz.org/obr/releases/repository.xml

For Simple IGB App, the URI was:
* /media/simple-igb-app/releases/0.0.1/620697076759907610_simple-igb-app-0.0.1.jar

which was redirected to absolute URL:
* https://devappstore5-media-0000.s3.amazonaws.com/media/simple-igb-app/releases/0.0.1/620697076759907610_simple-igb-app-0.0.1.jar

When I hit the above link to the jar file, I got a "permission denied" error. However, the root cause wasn't that the file existed and was not world-readable. Actually, the root cause was that the endpoint above did not exist.

It appears that when the jar file was uploaded, the upload mechanism saved it to:
* https://devappstore5-media-0000.s3.amazonaws.com/media/simpleigbapp/releases/0.0.1/620697076759907610_simple-igb-app-0.0.1.jar

Maybe there was an inconsistency between App Store versions? I used the Django admin interface to delete all the apps, and I deleted the app folder in S3 using the AWS console. I then submitted and approved a new App.

The OBR index file used the Bundle-SymbolicName minutes hyphens, and this was also where it was stored in S3.

I would rather use the Bundle-SymbolicName as the folder name in S3 because it's more consistent with how jar repositories (e.g., maven) are organized. Maven puts jars into folders named for the group id, the artifact id, and the version. I think we should do that, too, even if thought this is not a maven repository.
]

Hide

Permalink

Ann Loraine added a comment - 03/Sep/19 1:36 PM

Sameer Shanbhag and I have both tested the new configuration - it is working as expected. Moving this forward to Closed.

Show

Ann Loraine added a comment - 03/Sep/19 1:36 PM Sameer Shanbhag and I have both tested the new configuration - it is working as expected. Moving this forward to Closed.