Details
-
Type:
New Feature
-
Status: Closed (View Workflow)
-
Priority:
Major
-
Resolution: Won't Fix
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:
-
Story Points:2
-
Epic Link:
-
Sprint:Summer 2019 Sprint 12, Fall 2019 Sprint 2, Fall 2019 Sprint 3
Description
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure.
Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things.
Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store.
We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket.
Jar files are rarely large, so we think this will not be too taxing on the server.
Notes:
We searched for: "proxy server for S3 in django" and found:
- https://stackoverflow.com/questions/2636783/django-as-s3-proxy
- https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 - READ THIS! See comment near the end which tells us that proxying this way is a terrible idea, and a comment on that comment that gives a counter-argument.
For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use.
We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. Links I found:
Example configuration from above:
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
ProxyRequests off
ProxyPass /s3/ http://s3.amazonaws.com/your_bucket/
Attachments
Activity
| Field | Original Value | New Value |
|---|---|---|
| Epic Link | IGBF-1388 [ 17463 ] |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Description |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy * https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. |
| Description |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy * https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy * https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 - READ THIS! For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. |
| Description |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy * https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 - READ THIS! For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy * https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 - READ THIS! See comment near the end which tells us that proxying this way is a terrible idea. For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. |
| Description |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy * https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 - READ THIS! See comment near the end which tells us that proxying this way is a terrible idea. For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy * https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 - READ THIS! See comment near the end which tells us that proxying this way is a terrible idea, and a comment on that comment that argues a different way. For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. |
| Description |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy * https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 - READ THIS! See comment near the end which tells us that proxying this way is a terrible idea, and a comment on that comment that argues a different way. For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy * https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 - READ THIS! See comment near the end which tells us that proxying this way is a terrible idea, and a comment on that comment that gives a counter-argument. For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. |
| Description |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy * https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 - READ THIS! See comment near the end which tells us that proxying this way is a terrible idea, and a comment on that comment that gives a counter-argument. For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. |
Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.
However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure. Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things. Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store. We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket. Jar files are rarely large, so we think this will not be too taxing on the server. Notes: We searched for: "proxy server for S3 in django" and found: * https://stackoverflow.com/questions/2636783/django-as-s3-proxy * https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584 - READ THIS! See comment near the end which tells us that proxying this way is a terrible idea, and a comment on that comment that gives a counter-argument. For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use. We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. Links I found: * https://www.mattburkedev.com/apache-proxy-for-amazon-s3/ Example configuration from above: LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module modules/mod_proxy_http.so ProxyRequests off ProxyPass /s3/ http://s3.amazonaws.com/your_bucket/ |
Sure I can do that
[~aloraine]
I came across the following resource on the Amazon Bucket Policy Example Documentation:
https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html
There is a policy named aws:Referer which can be used to allow access to the requests coming from a specific domain and it will work with our current settings.
Update:
aws:Referer is probably not the best thing to do as the referer policy can be passed by a basic curl command as follows:
I think we will have to use proxy for jars.
For images, we can use redirect. No-one should ever try to grab one of the images stored in S3 unless the image is link is on an appstore page.
Can we:
- Configure apache to do redirection for everything except jar files, for which we will use proxy?
[~aloraine]
I was looking through different implementation and serving Media files from S3 using proxy seems to be the best way to do it. I am currently working on that.
For images currently, it requires the folder in the bucket to be public, a redirect will not work, as the images are handled by Django as a part of its implementation where the image is directly served from S3, we will have to figure out a way to serve those as well as a part of this ticket.
| Sprint | Summer 2019 Sprint 12 [ 71 ] | Summer 2019 Sprint 12, Fall 2019 Sprint 1 [ 71, 72 ] |
| Rank | Ranked higher |
| Status | In Progress [ 3 ] | Open [ 1 ] |
| Workflow | Loraine Lab Workflow [ 18695 ] | Fall 2019 Workflow Update [ 19021 ] |
| Assignee | Sameer Shanbhag [ sameer ] |
| Status | Open [ 1 ] | To-Do [ 10305 ] |
| Status | To-Do [ 10305 ] | In Progress [ 3 ] |
| Status | In Progress [ 3 ] | To-Do [ 10305 ] |
| Sprint | Summer 2019 Sprint 12, Fall 2019 Sprint 1 [ 71, 72 ] | Summer 2019 Sprint 12, Fall 2019 Sprint 2 [ 71, 73 ] |
| Sprint | Summer 2019 Sprint 12, Fall 2019 Sprint 2 [ 71, 73 ] | Summer 2019 Sprint 12, Fall 2019 Sprint 2, Fall 2019 Sprint 4 [ 71, 73, 74 ] |
| Rank | Ranked higher |
We decided to configure S3 on creation to allow public access of Apps. Documented same in documentation. Closing this issue as the solution is working fine.
| Status | To-Do [ 10305 ] | In Progress [ 3 ] |
| Status | In Progress [ 3 ] | Needs 1st Level Review [ 10005 ] |
| Status | Needs 1st Level Review [ 10005 ] | First Level Review in Progress [ 10301 ] |
| Status | First Level Review in Progress [ 10301 ] | Ready for Pull Request [ 10304 ] |
| Status | Ready for Pull Request [ 10304 ] | Pull Request Submitted [ 10101 ] |
| Status | Pull Request Submitted [ 10101 ] | Reviewing Pull Request [ 10303 ] |
| Status | Reviewing Pull Request [ 10303 ] | Merged Needs Testing [ 10002 ] |
| Status | Merged Needs Testing [ 10002 ] | Post-merge Testing In Progress [ 10003 ] |
| Status | Post-merge Testing In Progress [ 10003 ] | Closed [ 6 ] |
| Assignee | Noor Zahara [ noor91zahara ] |
| Assignee | Noor Zahara [ noor91zahara ] | Sameer Shanbhag [ sameer ] |
| Resolution | Won't Fix [ 2 ] |
| Workflow | Fall 2019 Workflow Update [ 19021 ] | Revised Fall 2019 Workflow Update [ 22482 ] |
From what I've read (not tons) this seems like the best solution:
In this solution, looks like we can use mod_proxy in Apache, thus require no coding changes for App Store, just server configuration.
Sameer Shanbhag - could you try it out the above on your DevAppStore? Looks like you would only need to make changes to default-ssl.conf and/or 000-default.conf