• name: Create a temporary snapshot for safety of RDS Host
  rds_snapshot:
  db_instance_identifier: "{{ db_instance_id }}"
  db_snapshot_identifier: "{{ ec2_name }}-snapshot-before-deployment"
  aws_access_key: "{{ AWS_ACCESS_KEY_ID }}"
  aws_secret_key: "{{ AWS_SECRET_ACCESS_KEY }}"
  region: "{{ rds_region }}"

The above code is used to create a snapshot of RDS

Please Review: https://bitbucket.org/chesterdias/chester-local-bioviz-connect-playbooks/branch/IGBF-2403#diff

Tasks Covered so far:
1. Redis server installation and setup
2. Setting.ini and venv setup
3. Creation of RDS host if not present
4. Setup of mysql database and creation of tables
5. Readme added for noting some steps in execution

As discussed with Nowlan Freese they do not run migration steps, instead manually setup the DB
As discussed with Chaitanya Kintali the use of DB is minimum since most dependency is on API

Chester Dias: Could you ask Chaitanya Kintali to do the code review and the functional review?

Chaitanya Kintali
Please verify and Review the tasks mentioned as completed.

Moving this back to TODO state for correction of some errors

Chaitanya Kintali
Please verify and Review the tasks mentioned as completed.

As per Discussion,

We no longer need creating new RDS instances, Since BioViz Connect has MySQL installed locally. Chester Dias Could you please modify the current playbooks to mimic the same behavior. Create local MySQL instance and follow the Cyverse Documentation for any help.
Moving to TODO.

Mysql Changes to create local mysql have been made
RDS role is removed

Note: For double best review, try standing up an all-new bioviz connect on a host in your own personal AWS account.

Chester Dias,

In the role Deploy_certificates and tasks main.yml, the certificate name is surrounded by " " which is looking for file "*.crt" and throwing an error. I guess the possible fix for this is to remove the "" double quotes from the main.yml from name variables and allow the system to set the string without any double quotes around it.

Chaitanya Kintali I have made the changes and removed the double quotes from the playbooks.

Chester Dias
It seems settings.ini values and keys are not present in secrets.yml and it could not find the key during setup.
Please refer to the cyverse documentation settings.ini to find all the possible config keys.

I have added the details to the secrets.yml to and made changes to the template to automatically get all the values into the settings.ini

Chester Dias

Still Facing the issue, Certificates not found even when the certificates are in the correct folders. attaching the screenshot.

error Certificates not found

I have changed the timeout for the certificates so you have more time for copying the certificates

Chaitanya Kintali Please let me know what is missing here to get the app running since now the apache too runs.

Chester Dias

Certificates are now getting copied in the correct folder and the ssl configuration is intact and matching with the current running ec2 configurations.
Redis is installed correctly.
All the tables are intact and created with the correct schema.

Note:
Both the SSLCertificateFile , SSLCertificateChainFile are copied with right names, but SSLCertificateKeyFile is copied as localhost.key. However, we can rename this later after setup.

Closing the ticket

Corrected the certificate file names and submitted PR:https://bitbucket.org/lorainelab/bioviz-connect-playbooks/pull-requests/3/igbf-2403/diff

Requesting confirmation from Chaitanya Kintali and Chester Dias before merging latest PR:

• If I run the playbooks as-is and provide the certs in ansible user's home directory, will I get a properly functioning bioviz connect site – with certificates deployed correctly and with the correct names?

[~aloraine]
Yes
The names of certificate files should be mentioned in the group_vars/common.yml
During the certificate deployment step, there is an automatic wait triggered to give the user time to upload the certificate files on the Bioviz Connect instance that is created.
The certificates should be placed on the Bioviz connect EC2 home directory after it is provisioned.

If ansible detects the presence of the files during this wait, it will copy the files to the correct directory

Chaitanya Kintali
Can you please confirm we have the Bioviz connect setup fully running after the execution of the playbook?

I actually prefer a different approach that will be easier for me!

Ask the user to place the certs in the home directory of the ansible user on the control node. Do not require the user to upload them manually to the target node.

Do please use bioviz-playbook as a model for how this works.

I am seeing the apache page after hitting the ec2 URL once the setup is done.
I am guessing the reason behind this is
1) domain name
2) some issue with wsgi. attaching the error file.

Error after changing the rewrite ip address to ec2 ip.

I was able to get some of the issues fixed and reach to the point of cas authentication on the server link
Chaitanya Kintali whenever u get some time u can pull the latest changes and follow the below steps to view the application running

sudo nano /var/www/vhosts/django_cyversedev/django_cyversedev/settings.ini

change ROOT_URL to ec2 host

sudo nano /etc/apache2/sites-available/000-default.conf

change SERVER_NAME =<add ec2 host url>

sudo systemctl restart apache2

Chester Dias

Below are the issues that I came across and after fixing them directly in the new ec2-instance I am able to see the application running.

1) Keep the database name defined in a separate variable as "BioVizCyverse" since the same name is being used in scripts and settings.py
2) Also for some reason, the database user doesn't seem to be created. I manually created the database user and then granted permissions and then it worked.
3) And there have been some changes in settings.ini file. The newly created ec2 doesn't have the updated changes from settings.ini
4) Assign ROOT_URL, ALLOWED_HOSTS, REDIRECT_URI, REDIRECT_URI_TIMEOUT to the domain name variable set in the playbook. Please refer settings.ini to see the values for the current system.

Please let me know once the above changes are done.As per my analyses, the application should run directly after running the playbook once the above changes are done.

Chaitanya Kintali
Thanks for testing this out. I have made the requested changes.
The MySQL user was created but the hostname was pointing to the IP address of the server rather than localhost like the way you had mentioned earlier, I have corrected the issue after verifying from karthiks instance of EC2,
The Redis password issue too has been removed.
I have changed default DB name to BioVizCyverse.

Please pull the latest changes and let me know if anything else needs to be amended.

Chester Dias

Finally Working
Moving to Ready for a Pull Request.

PR:https://bitbucket.org/lorainelab/bioviz-connect-playbooks/pull-requests/4/igbf-2403/diff

Request for Chaitanya Kintali, Chester Dias, and/or Karthik Raveendran:

Please add comments to example_secrets.yml explaining how to fill in these values:

# Redis Credentials
redis_pass: NotRealPWD

# Content of the settings.ini
secret_key: NOTAREALVALUE
username: NOTAREALVALUE
pwd: NOTAREALVALUE
access_token: NOTAREALVALUE
client_id: NOTAREALVALUE
client_secret: NOTAREALVALUE

Please add comments explaining which of the above values can I make up and which need to be obtained by somehow registering the site with CyVerse?

Hi Chaitanya Kintali, Karthik Raveendran, Nowlan Freese
I have added some comments about some variables. However, I am unsure about below ones. Could someone let me know if these are needed, If not then I can replace the settings.ini template in the playbooks with a new one without these properties entirely.

# ?Unsure of this values requirement
secret_key: NOTAREALVALUE

# ?Unsure of this values requirement
username: NOTAREALVALUE

# Password ?Unsure of this values requirement.
pwd: NOTAREALVALUE

# ?Unsure of this values requirement
access_token: NOTAREALVALUE

 

https://bitbucket.org/chesterdias/chester-local-bioviz-connect-playbooks/commits/ae31d1089011274af3f12460f139f2beb6e884e9

CC:[~aloraine]

Change request for Chester Dias:

• Please remove "Notarealvalue" text.

Also, a request for Chester Dias: Could you schedule a time with Nowlan Freese to get the values that are needed for these variables?

I can't attempt to run the playbook until I know what to enter in place of client_id, access_token, secret_key, etc. Some of them – such as redis_pass and secret_key look like values I can make up myself, but access_token, client_id, and client_secret look like values I need to get directly from Dr. Freese.

Can I use the values from one of the deployed bioviz connect sites?

Please advise.

Requested changes made.
Please review: https://bitbucket.org/chesterdias/chester-local-bioviz-connect-playbooks/branch/IGBF-2403#diff

Please submit PR. I will merge and then either myself or Nowlan Freese will attempt to run the playbooks.

PR: https://bitbucket.org/lorainelab/bioviz-connect-playbooks/pull-requests/5/igbf-2403/diff

Request for Chester Dias:

Please investigate why pull request is not being shown as merged in the Jira ticket. The PR appears on the right side of other tickets. Why not this one?

Can you check the below-mentioned settings and see if you can see the repository 'bioviz-connect-playbooks' being listed.
Login as admin into JIRA
Administration 
Click on Add-ons tab
Click on DVCS Accounts in the left side panel
I don't see the repository('bioviz-connect-playbooks') listed in the replicated account. I think this has something to do with the bitbucket user configured in JIRA (to access bitbucket) accessing a private repository.
Please let me know if this solves the issue.

Cc: [~aloraine]

Thanks for the explanation. I see the problem now!

I would like for the playbooks repository to be public, but I'm worried we have already inadvertently exposed variables that cannot be public - such as, the authorization tokens sent by the bioviz connect site to cyverse.

Chester Dias could you investigate with with help from Chaitanya Kintali and/or Nowlan Freese?

If those variables have been exposed, we either should expire them and create new ones or create an entirely new playbooks repository without the exposed variables. In either case, we need to do a kind of an audit of all the variables captured in these playbooks.

Lastly, for testing, Nowlan Freese - could you help with this? We ought to try to run the playbooks from start to finish using an AWS account.

I will check to see if any access tokens have been exposed at any commit point.
Also [~aloraine] I noticed the account used for JIRA DVCS settings was a personal JIRA account. Perhaps we should use a generic bot account.

[~aloraine] I don't see any sensitive variable exposed in any commits as such.

Thank you. Making the repository public.

Suggestions on how to test:

• Stand up a working BioViz Connect in your AWS account.
• Check that you can use a non-bioviz domain certificate
• Check that there are instructions on how to set up the call-back from CyVerse

Need to install git on the control node: sudo yum install git

After the ansible playbook had completed running and I had bound the ip address to the correct domain (https://nowlantest.bioviz.org) I was still not able to connect. After restarting apache manually (sudo service apache2 restart) I was able to login. Once logged in there was a problem with the redis server which needed one of the sockets exposed (sudo chmod 777 /run/redis/redis-server.sock). The login process was completed and data were loaded. However, I needed to sync the apps database before I could run analyses:

curl -L -X GET 'https://nowlantest.bioviz.org/SyncAppDataToBioViz/?accesstoken=AT-7417-wTHnOYWWjdI6x4XZadgv-CLEKDBJD0gt'

Syncing the apps database is somewhat tricky as it currently requires a live access token. This is something that Chaitanya Kintali and I have discussed changing.

Missing tag in commons.yml for ec2_stack_tag. Had to run playbook with ansible-playbook main.yml --extra-vars "ec2_stack_tag=dev"

I have a shell script that can fetch the token and put it in a text file. maybe this step too can be included in the playbooks
curl -L -X GET 'https://nowlantest.bioviz.org/SyncAppDataToBioViz/?accesstoken=AT-7417-wTHnOYWWjdI6x4XZadgv-CLEKDBJD0gt'

Why do we need 777 privileges on the redis server socket? Exposing a file leaves it prone to changes from any internal or external process from any user including the guest account

Nowlan FreeseThe login issue I will look into it along with the ec2_stack_tag issue

I've asked Chaitanya Kintali to comment on the Redis socket issue.

I think it would be good to include a script to hit the SyncAppDataToBioViz endpoint. The SyncAppDataToBioViz endpoint needs to be hit every time there is a new release. Would need to use the CyVerse username/password to first get an access token.

Replying to Chester Dias, on redis socket 777 issue,

The socket must be accessible for www-data. Hence we are modifying the socket to have permissions so that Redis group can access it. Chester Dias, please let us know if there is any way around

Based on my discussion and trying the new permissions with Karthik Raveendran help
I think it will be best if we do the following with respect to the socket file
chown redis:www-data /run/redis/redis-server.sock
chmod 770 /run/redis/redis-server.sock

This will restrict the access of the permission to redis user and www-data group.

I am making the required change in the playbooks

I have added the missing tag ec2_stack_tag
For the apache restart, Seems like I had written the code but never invoked it. I have added the invocation now.
I have also added a new role for syncing the data using a shell script. Please review the script too. The script name is sync_app_data.sh.j2 the attributes in curly braces will be replaced at runtime with the appropriate value from the secrets.yml
Nowlan Freese Please review: https://bitbucket.org/chesterdias/chester-local-bioviz-connect-playbooks/branch/IGBF-2403#diff

The script looks good, it worked on my local system.

You do not need to include the terrain_user and terrain_pass variables. The username and pwd variables in the example_secrets.yml are what should be used as they contain the CyVerse username and password required for obtaining the access token.

Nowlan Freese - Do "CyVerse username and password" need to be secret?

[~aloraine] - they do need to be secret.

Request for Nowlan Freese:

I would like to close this ticket. Any bug fixes or improvements to the playbooks can be added to a new ticket. If you agree, please go ahead and move this to "Done".

Chester has a pending branch under IGBF-2403 that has yet to be merged. It may be confusing to create a new issue to merge that branch. I would like Chester to finish the last change I have added in the comments and then issue a pull request. Any new issues found after will be put under a new ticket number.

Nowlan Freese I have removed the extra variables as per the comments. Please review the change: https://bitbucket.org/chesterdias/chester-local-bioviz-connect-playbooks/branch/IGBF-2403#diff

Chester Dias changes look good, go ahead and create a pull request.

https://bitbucket.org/lorainelab/bioviz-connect-playbooks/pull-requests/8/igbf-2403/diff

PR:https://bitbucket.org/lorainelab/bioviz-connect-playbooks/pull-requests/9/igbf-2403-added-a-change-in-interpreter-to/diff

Merged.

Requires a final restart of apache due to switching of the ip address.

Please review: https://bitbucket.org/chesterdias/chester-local-bioviz-connect-playbooks/commits/198686143f5a4202fa5f9cad48ebcaef461dab5a?at=IGBF-2403

Chester Dias go ahead and do a pull request and I will test as soon as it's merged.

[~aloraine] do you want the playbook to update the EC2 to do a git pull of the most recent version of the BioViz Connect bitbucket repo?

The situation I'm thinking of is that the playbook creates the BioViz Connect EC2. Time passes. The EC2 is now behind master branch of BioViz Connect. Do you rerun the playbook and expect it to pull down the most recent changes, or do you log in to the BioViz Connect EC2 and manually pull the changes?

I advise using the ansible "git" module to pull the latest commit for whatever branch you have deployed.

A couple things to keep in mind:

• If there are any edits to version-controlled files in your cloned copy (on the server), then those edits will be lost.
• If there are files saved in the cloned repo residing on your server that are NOT in the repository, such as something a user uploaded, then those will NOT be lost.

As an example of how this can work, see the bioviz-playbooks.

We configured it to let us replace the cloned copy on the server with newer versions, with different branches, with different branches from different repositories, etc.

You can see how this is working by looking at the "clone" role:

https://bitbucket.org/lorainelab/bioviz-playbooks/src/master/roles/clone/tasks/main.yml

PR: https://bitbucket.org/lorainelab/bioviz-connect-playbooks/pull-requests/10/igbf-2403/diff

Merged.

Need one final apache restart for the site to be functional. There is a final restart in the playbook, but it doesn't seem to be working. Chester is going to come up with a work around.

Created IGBF-2571 to address the issue of the final apache restart.

Closing issue.