Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-1718

Plan how to use S3 to store and serve App images, logo, & jars

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:

      Description

      Currently, when a user submits an App, images get stored in the deployed project directory.

      For better robustness, it would be better to store digital assets uploaded by users in S3 or other cloud storage that has lifecycle independent from the VM where App Store is deployed.

      To do:

      • Investigate whether someone has already created a Django App for retrieving/storing/uploading images or other assets to S3
      • Plan how we will implement S3-based (cloud) storage for images in App Store

        Attachments

          Activity

          Hide
          ann.loraine Ann Loraine added a comment - - edited

          Tutorials showing how to use s3 for static files in Django.

          Also includes docker and docker compose:

          Uses Django App django-storages. Can use this with dropbox - not just s3.

          Show
          ann.loraine Ann Loraine added a comment - - edited Tutorials showing how to use s3 for static files in Django. Also includes docker and docker compose: https://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/ Uses Django App django-storages. Can use this with dropbox - not just s3. https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-project.html
          Hide
          ann.loraine Ann Loraine added a comment -
          Show
          ann.loraine Ann Loraine added a comment - A caution : Some people think Docker is terrible: http://www.smashcompany.com/technology/docker-is-a-dangerous-gamble-which-we-will-regret
          Hide
          ann.loraine Ann Loraine added a comment - - edited

          It would be nice-to-have (but not essential) if we can control who gets to download assets from our S3. This would allow us to protect against bad actors downloading everything from our bucket over and over and raising costs.
          For example:
          Maybe there is a way to configure links to digital assets so that our Django gets them from S3 and then sends the data to the user's browser.

          Show
          ann.loraine Ann Loraine added a comment - - edited It would be nice-to-have (but not essential) if we can control who gets to download assets from our S3. This would allow us to protect against bad actors downloading everything from our bucket over and over and raising costs. For example: Maybe there is a way to configure links to digital assets so that our Django gets them from S3 and then sends the data to the user's browser.
          Hide
          ann.loraine Ann Loraine added a comment - - edited

          Note: Django has tooling for configuring site-specific variables that should not be exposed, e.g., App passwords. See:

          I think we are now at the point where we need to develop a principled and uniform way to enable individual users/sites to configure credentials.

          Show
          ann.loraine Ann Loraine added a comment - - edited Note : Django has tooling for configuring site-specific variables that should not be exposed, e.g., App passwords. See: https://pypi.org/project/python-decouple/ I think we are now at the point where we need to develop a principled and uniform way to enable individual users/sites to configure credentials.
          Hide
          svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment -

          [~aloraine]

          Steps to serve media-content/jars using S3.

          1. Install boto3 and django-storages

          pip install boto3
          pip install django-storages

          2. In the INSTALLED_APPS array add 'storages' in settings.py

          3. Add below lines to settings.py

          AWS_ACCESS_KEY_ID = 'YOUR ACCESS KEY ID'
          AWS_SECRET_ACCESS_KEY = 'YOUR SECRET KEY'
          AWS_STORAGE_BUCKET_NAME = 'YOUR BUCKET NAME'
          AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
          AWS_S3_OBJECT_PARAMETERS =

          { 'CacheControl': 'max-age=86400', }

          AWS_LOCATION = 'media'

          4. Remove previos URL setting and add below lines

          MEDIA_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)
          DEFAULT_FILE_STORAGE = 'appstore.storage_backends.MediaStorage'

          5. Create a class Mediastorage and paste the below code. (same locations as settings.py)

          storage_backends.py

          Code:
          from storages.backends.s3boto3 import S3Boto3Storage

          class MediaStorage(S3Boto3Storage):
          location = 'media'
          file_overwrite = False

          Steps to Maintain Credentials using Python Decouple

          1. pip install python-decouple

          2. create settings.ini file in appstore/appstore
          EMAIL_HOST_PASSWORD=xxxxxxx
          SOCIAL_AUTH_GOOGLE_OAUTH2_KEY=xxxxxxx
          SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET=xxxxxxx
          AWS_ACCESS_KEY_ID=xxxxxxx
          AWS_SECRET_ACCESS_KEY=xxxxxxx

          3. import config:
          --> from decouple import config
          --> get creds using

          Show
          svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - [~aloraine] Steps to serve media-content/jars using S3. 1. Install boto3 and django-storages pip install boto3 pip install django-storages 2. In the INSTALLED_APPS array add 'storages' in settings.py 3. Add below lines to settings.py AWS_ACCESS_KEY_ID = 'YOUR ACCESS KEY ID' AWS_SECRET_ACCESS_KEY = 'YOUR SECRET KEY' AWS_STORAGE_BUCKET_NAME = 'YOUR BUCKET NAME' AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME AWS_S3_OBJECT_PARAMETERS = { 'CacheControl': 'max-age=86400', } AWS_LOCATION = 'media' 4. Remove previos URL setting and add below lines MEDIA_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION) DEFAULT_FILE_STORAGE = 'appstore.storage_backends.MediaStorage' 5. Create a class Mediastorage and paste the below code. (same locations as settings.py) storage_backends.py Code: from storages.backends.s3boto3 import S3Boto3Storage class MediaStorage(S3Boto3Storage): location = 'media' file_overwrite = False Steps to Maintain Credentials using Python Decouple 1. pip install python-decouple 2. create settings.ini file in appstore/appstore EMAIL_HOST_PASSWORD=xxxxxxx SOCIAL_AUTH_GOOGLE_OAUTH2_KEY=xxxxxxx SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET=xxxxxxx AWS_ACCESS_KEY_ID=xxxxxxx AWS_SECRET_ACCESS_KEY=xxxxxxx 3. import config: --> from decouple import config --> get creds using
          Hide
          svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment -

          [~aloraine]

          Changed the location of S3 storage from my account to lorainelab account.

          Added documentation to Appstore google doc.

          Everything is working as expected.

          Show
          svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - [~aloraine] Changed the location of S3 storage from my account to lorainelab account. Added documentation to Appstore google doc. Everything is working as expected.
          Hide
          ann.loraine Ann Loraine added a comment -

          To test:

          • Stand up a VM on EC2
          • Get the latest code
          • Create your own S3 via your own account
          • Add a new fake App and make sure digital assets get stored in the S3 bucket: jars, screen shots, logos
          Show
          ann.loraine Ann Loraine added a comment - To test: Stand up a VM on EC2 Get the latest code Create your own S3 via your own account Add a new fake App and make sure digital assets get stored in the S3 bucket: jars, screen shots, logos
          Show
          sameer Sameer Shanbhag (Inactive) added a comment - Please refer to the following document : https://docs.google.com/document/d/1_9C03q6TD5wjLqfVLKsuDsIEQ4-qrS0JhONO5VBOwaA/edit?ts=5ce56b5e
          Hide
          ann.loraine Ann Loraine added a comment - - edited

          Side question:

          • Can different app logos co-exist? For example, can the same App change logo image between versions?
          Show
          ann.loraine Ann Loraine added a comment - - edited Side question: Can different app logos co-exist? For example, can the same App change logo image between versions?
          Hide
          svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment -

          [~aloraine]

          I think there is no such code which supports multiple logos. But it is possible to do.

          Show
          svallapu Sai Charan Reddy Vallapureddy (Inactive) added a comment - [~aloraine] I think there is no such code which supports multiple logos. But it is possible to do.

            People

            • Assignee:
              svallapu Sai Charan Reddy Vallapureddy (Inactive)
              Reporter:
              ann.loraine Ann Loraine
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: