Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-1971

Serve content from media by proxy not redirect

    Details

    • Type: New Feature
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:

      Description

      Currently, we are using redirect rules in Apache in order to serve jar and logo files from an AppStore's S3 bucket.

      However, this is problematic. For this to work, permissions need to be set on the bucket that make it less secure.

      Instead, we would like to control all access to the S3 bucket. This is important to control costs, among other things.

      Goal: Only AppStore itself can get data from the S3 bucket. We want clients (e.g., IGB) to be able to download artifacts from the bucket - such as App jar files. But we only want this to happen through App Store.

      We have discussed it and think the best solution will be to configure AppStore to function as a proxy. When a client requests an artifact that is actually stored in S3, AppStore will authenticate to the bucket and stream the data through itself and to the client. The client will never know where the data actually came from. No-one will ever see the address of our S3 bucket.

      Jar files are rarely large, so we think this will not be too taxing on the server.

      Notes:

      We searched for: "proxy server for S3 in django" and found:

      For this task, we should read above and also do more searching as there may already be some very cool libraries already available for us to use.

      We should also investigate whether we can get Apache to handle the proxy aspect. If we can configure Apache in a different way, this will save us some headaches of coding. Links I found:

      Example configuration from above:

      LoadModule proxy_module modules/mod_proxy.so
      LoadModule proxy_http_module modules/mod_proxy_http.so
      ProxyRequests off
      ProxyPass /s3/ http://s3.amazonaws.com/your_bucket/

        Attachments

          Activity

          Hide
          ann.loraine Ann Loraine added a comment - - edited

          From what I've read (not tons) this seems like the best solution:

          In this solution, looks like we can use mod_proxy in Apache, thus require no coding changes for App Store, just server configuration.

          Sameer Shanbhag - could you try it out the above on your DevAppStore? Looks like you would only need to make changes to default-ssl.conf and/or 000-default.conf

          Show
          ann.loraine Ann Loraine added a comment - - edited From what I've read (not tons) this seems like the best solution: https://forums.aws.amazon.com/thread.jspa?threadID=20087 In this solution, looks like we can use mod_proxy in Apache, thus require no coding changes for App Store, just server configuration. Sameer Shanbhag - could you try it out the above on your DevAppStore? Looks like you would only need to make changes to default-ssl.conf and/or 000-default.conf
          Hide
          sameer Sameer Shanbhag (Inactive) added a comment -

          Sure I can do that

          Show
          sameer Sameer Shanbhag (Inactive) added a comment - Sure I can do that
          Hide
          sameer Sameer Shanbhag (Inactive) added a comment - - edited

          [~aloraine]

          I came across the following resource on the Amazon Bucket Policy Example Documentation:

          https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html

          There is a policy named aws:Referer which can be used to allow access to the requests coming from a specific domain and it will work with our current settings.

          Update:
          aws:Referer is probably not the best thing to do as the referer policy can be passed by a basic curl command as follows:

          https://security.stackexchange.com/questions/135677/security-of-setting-access-to-s3-bucket-only-from-a-particular-website/135706#135706

          Show
          sameer Sameer Shanbhag (Inactive) added a comment - - edited [~aloraine] I came across the following resource on the Amazon Bucket Policy Example Documentation: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html There is a policy named aws:Referer which can be used to allow access to the requests coming from a specific domain and it will work with our current settings. Update: aws:Referer is probably not the best thing to do as the referer policy can be passed by a basic curl command as follows: https://security.stackexchange.com/questions/135677/security-of-setting-access-to-s3-bucket-only-from-a-particular-website/135706#135706
          Hide
          ann.loraine Ann Loraine added a comment -

          I think we will have to use proxy for jars.
          For images, we can use redirect. No-one should ever try to grab one of the images stored in S3 unless the image is link is on an appstore page.

          Can we:

          • Configure apache to do redirection for everything except jar files, for which we will use proxy?
          Show
          ann.loraine Ann Loraine added a comment - I think we will have to use proxy for jars. For images, we can use redirect. No-one should ever try to grab one of the images stored in S3 unless the image is link is on an appstore page. Can we: Configure apache to do redirection for everything except jar files, for which we will use proxy?
          Hide
          sameer Sameer Shanbhag (Inactive) added a comment -

          [~aloraine]

          I was looking through different implementation and serving Media files from S3 using proxy seems to be the best way to do it. I am currently working on that.

          For images currently, it requires the folder in the bucket to be public, a redirect will not work, as the images are handled by Django as a part of its implementation where the image is directly served from S3, we will have to figure out a way to serve those as well as a part of this ticket.

          Show
          sameer Sameer Shanbhag (Inactive) added a comment - [~aloraine] I was looking through different implementation and serving Media files from S3 using proxy seems to be the best way to do it. I am currently working on that. For images currently, it requires the folder in the bucket to be public, a redirect will not work, as the images are handled by Django as a part of its implementation where the image is directly served from S3, we will have to figure out a way to serve those as well as a part of this ticket.
          Hide
          ann.loraine Ann Loraine added a comment -

          We decided to configure S3 on creation to allow public access of Apps. Documented same in documentation. Closing this issue as the solution is working fine.

          Show
          ann.loraine Ann Loraine added a comment - We decided to configure S3 on creation to allow public access of Apps. Documented same in documentation. Closing this issue as the solution is working fine.

            People

            • Assignee:
              sameer Sameer Shanbhag (Inactive)
              Reporter:
              ann.loraine Ann Loraine
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: