Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-2087

Implement data cleaning mechanism to remove stale entries from AppPending table and S3 bucket

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
    • Story Points:
      1
    • Sprint:
      Fall 5 : 14 Oct to 25 Oct, Fall 6 : 28 Oct to 8 Nov, Fall 7 : 11 Nov to 22 Nov, Fall 8 : 25 Nov to 6 Dec, Fall 9 : 9 Dec to 20 Dec

      Attachments

        Issue Links

          Activity

          Hide
          ann.loraine Ann Loraine added a comment -

          Make sure to update python_packages.txt with any new modules required. See commits for how to use pip freeze to make sure all dependencies are listed.

          Show
          ann.loraine Ann Loraine added a comment - Make sure to update python_packages.txt with any new modules required. See commits for how to use pip freeze to make sure all dependencies are listed.
          Hide
          rpatil14 Riddhi Jagdish Patil (Inactive) added a comment - - edited

          Issue:
          Implement data cleaning mechanism to remove stale entries from AppPending table and S3 bucket

          Solution:
          To delete the stale entries from the AppPending table and the jar from the S3 bucket implemented a scheduler which will run every midnight and delete the entries from the AppPending table for the entries with submitter_approval="False".

          Use below website to get the cronstring of the job if required to be changed it in future:
          https://crontab.guru/#0_0_*_*_*

          Files changed:
          Code changes are present in branch https://bitbucket.org/rpatil14/appstore_riddhi/branch/IGBF-2087#diff
          python_packages.txt
          submit_app/app_pending_cleanup.py
          submit_app/apps.py

          Testing steps:
          Scenario 1:
          1) Click on submit your app
          2) Select an app to submit from "Choose File" option or Enter URL
          3) Go back to the home page or click browser back button without submitting the app
          4) Entry will be created in the AppPending table with submitter_approved value as "False" and the jar file will be uploaded to the s3 bucket
          Expected output: Once the scheduler run at midnight this entry will be deleted from the AppPending table and the jar file will be deleted from the s3 bucket

          Scenario 2:
          1) Click on submit your app
          2) Select an app to submit from "Choose File" option or Enter URL
          3) Go back to the home page or click browser back button without submitting the app
          4) Repeat step 1-2 for the same app with same version and this time submit the app by clicking "It's all good. Submit my app" button
          5) Two entries should be created in the AppPending table with one entry submitter_approved value as "False" and other submitter_approved value as "True". Only one jar should be uploaded to the s3 bucket with the latest submission timestamp
          Expected output: Once the scheduler run at midnight entry submitter_approved value as "False" will be deleted from the AppPending table and the jar file will not be deleted from the s3 bucket

          Testing results:
          Code changes are deployed to https://dev-appstore-6.bioviz.org. Admin username and password for devappstore6 is same as devappstore2

          Note:
          Run command "pip install -r python_packages.txt" to install new package "apscheduler" required for this task

          cc.[~aloraine]

          Show
          rpatil14 Riddhi Jagdish Patil (Inactive) added a comment - - edited Issue: Implement data cleaning mechanism to remove stale entries from AppPending table and S3 bucket Solution: To delete the stale entries from the AppPending table and the jar from the S3 bucket implemented a scheduler which will run every midnight and delete the entries from the AppPending table for the entries with submitter_approval="False". Use below website to get the cronstring of the job if required to be changed it in future: https://crontab.guru/#0_0_*_*_* Files changed: Code changes are present in branch https://bitbucket.org/rpatil14/appstore_riddhi/branch/IGBF-2087#diff python_packages.txt submit_app/app_pending_cleanup.py submit_app/apps.py Testing steps: Scenario 1: 1) Click on submit your app 2) Select an app to submit from "Choose File" option or Enter URL 3) Go back to the home page or click browser back button without submitting the app 4) Entry will be created in the AppPending table with submitter_approved value as "False" and the jar file will be uploaded to the s3 bucket Expected output: Once the scheduler run at midnight this entry will be deleted from the AppPending table and the jar file will be deleted from the s3 bucket Scenario 2: 1) Click on submit your app 2) Select an app to submit from "Choose File" option or Enter URL 3) Go back to the home page or click browser back button without submitting the app 4) Repeat step 1-2 for the same app with same version and this time submit the app by clicking "It's all good. Submit my app" button 5) Two entries should be created in the AppPending table with one entry submitter_approved value as "False" and other submitter_approved value as "True". Only one jar should be uploaded to the s3 bucket with the latest submission timestamp Expected output: Once the scheduler run at midnight entry submitter_approved value as "False" will be deleted from the AppPending table and the jar file will not be deleted from the s3 bucket Testing results: Code changes are deployed to https://dev-appstore-6.bioviz.org . Admin username and password for devappstore6 is same as devappstore2 Note: Run command "pip install -r python_packages.txt" to install new package "apscheduler" required for this task cc. [~aloraine]
          Hide
          ann.loraine Ann Loraine added a comment - - edited

          A request:

          Can you modify the scheduler so that it only deletes "stale" apps that are more than 5 hours old?

          Show
          ann.loraine Ann Loraine added a comment - - edited A request: Can you modify the scheduler so that it only deletes "stale" apps that are more than 5 hours old?
          Hide
          rpatil14 Riddhi Jagdish Patil (Inactive) added a comment -

          Delete entries from app table which are 5 hours old.
          https://bitbucket.org/rpatil14/appstore_riddhi/branch/IGBF-2087#diff

          cc. [~aloraine]

          Show
          rpatil14 Riddhi Jagdish Patil (Inactive) added a comment - Delete entries from app table which are 5 hours old. https://bitbucket.org/rpatil14/appstore_riddhi/branch/IGBF-2087#diff cc. [~aloraine]
          Show
          rpatil14 Riddhi Jagdish Patil (Inactive) added a comment - pull request submitted: https://bitbucket.org/lorainelab/appstore/pull-requests/153/igbf-2087/diff
          Hide
          ann.loraine Ann Loraine added a comment -

          To change the time span for deleting a stale app, edit: app_pending_cleanup.py
          Will make this configuration vai settings.py and settings.ini.template

          Show
          ann.loraine Ann Loraine added a comment - To change the time span for deleting a stale app, edit: app_pending_cleanup.py Will make this configuration vai settings.py and settings.ini.template
          Hide
          ann.loraine Ann Loraine added a comment -

          Merged into the master branch and deployed to https://dev-appstore-2.bioviz.org for testing.

          Show
          ann.loraine Ann Loraine added a comment - Merged into the master branch and deployed to https://dev-appstore-2.bioviz.org for testing.
          Hide
          ann.loraine Ann Loraine added a comment -

          Shamika Gajanan Kulkarni This has been sitting in the testing queue assigned to yourself for about a week. Please indicate the status - does it work and can you now close it?

          Show
          ann.loraine Ann Loraine added a comment - Shamika Gajanan Kulkarni This has been sitting in the testing queue assigned to yourself for about a week. Please indicate the status - does it work and can you now close it?
          Hide
          shamika Shamika Gajanan Kulkarni (Inactive) added a comment - - edited

          Prof [~aloraine] I have successfully tested the first scenario mentioned above. Today I started on the second and will update the status tomorrow as the scheduler will run at midnight. I should be able to close it then.

          Show
          shamika Shamika Gajanan Kulkarni (Inactive) added a comment - - edited Prof [~aloraine] I have successfully tested the first scenario mentioned above. Today I started on the second and will update the status tomorrow as the scheduler will run at midnight. I should be able to close it then.
          Hide
          shamika Shamika Gajanan Kulkarni (Inactive) added a comment -

          This ticket has been tested and I had observed that the jar for the app without submitter approval got deleted when the scheduler runs. I just want to make sure that it was done by the scheduler itself and not someone else who was working on Testing the Appstore tickets. Just to be sure I am testing this once again today. Hope it's okay.

          Show
          shamika Shamika Gajanan Kulkarni (Inactive) added a comment - This ticket has been tested and I had observed that the jar for the app without submitter approval got deleted when the scheduler runs. I just want to make sure that it was done by the scheduler itself and not someone else who was working on Testing the Appstore tickets. Just to be sure I am testing this once again today. Hope it's okay.
          Hide
          shamika Shamika Gajanan Kulkarni (Inactive) added a comment -

          Successfully tested the ticket. Observed the following:
          1) For the settings made recently, the scheduler runs every 2 hours [odd hours of the day like 13:00, 15:00, etc.] and the offset value is set to 1 hour.
          2) The entry for app with submitter_approved value as false in AppPending table is deleted if the app was uploaded before one hour time span. The jar is also deleted from S3 pending_releases. [Get Average Score was used to test this]
          3) When I tried to submit an app and returned to home page without clicking on Submit, an entry for the app was created in AppPending table with submitter_approved value as false. When I try to resubmit the same app and follow the entire process, there is only one entry in AppPending Table with submitter_approved value as True and the jar is replaced with the new timestamp in S3 pending_releases. When the scheduler runs, it does not delete this app entry or the jar from S3. [Merge Annotation Operator was used to test this]
          4) If an app is uploaded within the one hour time span, and the submitter_approved value is False, it does not get deleted by the scheduler right now as this app entry is not treated as stale entry yet. [Super Simple IGB app was used to test this.]

          Working as expected. Kindly refer the screenshots attached. Moving to Closed. Thank you.

          Show
          shamika Shamika Gajanan Kulkarni (Inactive) added a comment - Successfully tested the ticket. Observed the following: 1) For the settings made recently, the scheduler runs every 2 hours [odd hours of the day like 13:00, 15:00, etc.] and the offset value is set to 1 hour. 2) The entry for app with submitter_approved value as false in AppPending table is deleted if the app was uploaded before one hour time span. The jar is also deleted from S3 pending_releases. [Get Average Score was used to test this] 3) When I tried to submit an app and returned to home page without clicking on Submit, an entry for the app was created in AppPending table with submitter_approved value as false. When I try to resubmit the same app and follow the entire process, there is only one entry in AppPending Table with submitter_approved value as True and the jar is replaced with the new timestamp in S3 pending_releases. When the scheduler runs, it does not delete this app entry or the jar from S3. [Merge Annotation Operator was used to test this] 4) If an app is uploaded within the one hour time span, and the submitter_approved value is False, it does not get deleted by the scheduler right now as this app entry is not treated as stale entry yet. [Super Simple IGB app was used to test this.] Working as expected. Kindly refer the screenshots attached. Moving to Closed. Thank you.
          Hide
          ann.loraine Ann Loraine added a comment -

          Thank you Shamika Gajanan Kulkarni for the very clear & thorough description.

          Show
          ann.loraine Ann Loraine added a comment - Thank you Shamika Gajanan Kulkarni for the very clear & thorough description.

            People

            • Assignee:
              rpatil14 Riddhi Jagdish Patil (Inactive)
              Reporter:
              rpatil14 Riddhi Jagdish Patil (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: