Details
-
Type:
Task
-
Status: Closed (View Workflow)
-
Priority:
Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:None
-
Story Points:1.5
-
Epic Link:
-
Sprint:Summer 1: 8 Jun - 19 Jun, Summer 2: 22 Jun - 3 Jul, Summer 3: 6 Jul - 17 Jul, Summer 4: 14 Jul - 28 Jul, Summer 5: 3 Aug - 14 Aug, Summer 6: 17 Aug - 28 Aug, Summer 7: 31 Aug - 11 Sep, Fall 1: 14 Sep - 25 Sep, Fall 2: 28 Sep - 9 Oct
Description
When we stand up a new appstore with all new code, we often want to copy data (msyql database and s3 bucket contents) to a new database and new s3 bucket to be used by the new appstore.
Currently, we do not have a good way to copy the s3 bucket contents. (We recently developed code for copying the mysql database, however - see IGBF-2421.)
For this task, we'll write some code that copies data from an existing s3 bucket into a newly created one for the new ec2 to use.
Let's add some new tasks that accomplish this goal to the end of main.yml in role S3 in appstore playbooks.
The new tasks should check to see if a variable first_s3_bucket_name exists. This is to ensure that if the user is not actually trying to copy over some S3 contents, the tasks will not run.
If variable first_s3_bucket_name exists and first_s3_bucket_name does not equal s3_bucket_name and s3_bucket_name is empty, copy the contents of first_s3_bucket_name into s3_bucket_name.
Note: I'm entirely sure that the mysql database contains references to particular s3 buckets or if it is using relative paths of some type. Someone needs to check this. If it contains references to the s3 bucket name, we will need to add some extra logic to deal with that.
Attachments
Issue Links
- blocks
-
-
- Closed
-
Activity
copy is failing:
TASK [copy_s3 : Copy source bucket contents to destination (new) bucket] ***************************** fatal: [localhost]: FAILED! => changed=true cmd: aws s3 sync s3://testappstore-xyz s3://devappstore2-xyz delta: '0:00:00.595888' end: '2020-09-25 11:39:47.809347' msg: non-zero return code rc: 1 start: '2020-09-25 11:39:47.213459' stderr: 'fatal error: An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.' stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>
As noted in meeting today, attaching new IAM roles to ec2's is not working as expected.
Am going to try this instead:
- When creating the control node, I will assign it an IAM role and never delete this role
- When needed, I will temporarily attach the s3 "copy" policy and add it to the role
cc: Chester Dias
Above strategy worked. Needed to pause after attaching new policy to control node role.
Tested by standing up all-new appstore "devappstore3"
Got error on first try:
TASK [ec2 : Create devappstore3 if does not exist] **************************************************************** fatal: [localhost]: FAILED! => changed=false msg: 'Instance creation failed => InvalidParameterValue: Value (devappstore3) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name' PLAY RECAP ******************************************************************************************************** localhost : ok=19 changed=10 unreachable=0 failed=1 skipped=4 rescued=0 ignored=0
Worked on second try.
Tested replication of existing database and s3 bucket. Works. Moving to closed.
Suggestion: Look at Ansible Galaxy for code we can import that does this. I bet some-one has written something to do this. Basically, the job is to mirror an s3 bucket. Surely this has been written already!