As a general check to see if Terrain is running, can GET the endpoint: https://de.cyverse.org/terrain

If Terrain is running, it will return: The infinite is attainable with Terrain!

Tried doing wget and curl but unable to fetch the URL Status from the command line. Trying some more iterations

Total of 3 tools are needed
Grafana with world ping plugin
Telegraf agent can be used to monitor different attributes of the host machines and send data into influxdb.
Influxdb is the database used to hold all the data from telegraf.

Monitoring is tested to send out emails when the endpoint is found to be offline (Manual failure was tested). screenshots attached.

Sidenote for this issue:

Chester and I found that some EC2 images are blocked from reaching the Terrain endpoints (https://de.cyverse.org/terrain). The Ubuntu Server image currently used to host BioViz Connect is able to reach the Terrain endpoints, but the Amazon Linux 2 image is blocked. After talking with Sarah Roberts at CyVerse, our best guess is that CyVerse is blocking most, but not all, Amazon EC2 IP addresses.

Update from NF: Chester has been able to connect via a newly stood-up EC2 and will proceed using that.

Findings
SMTP configured email notifications are occurring using a fixed email ID.
Will need to configure the rest of the endpoints or write a simple shell script to hit the endpoint.

Attached configuration files of the grafana, influx, telegraf softwares. Also attached are screenshots for grafana board, Dashboard, and test email.

Below are the queries currently in use.
All endpoint responsetime

SELECT distinct("response_time") FROM "autogen"."http_response" WHERE $timeFilter GROUP BY time($__interval), "server"

Individual endpoint response time

SELECT "response_time" FROM "autogen"."http_response" WHERE ("server" ='$Endpoint') AND $timeFilter 

Status

SELECT "http_response_code" FROM "http_response" WHERE ("server" = '$Endpoint') AND $timeFilter

All Data

SELECT * FROM "autogen"."http_response" 

AWS AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190212.1 (ami-0a313d6098716f372)

We can proceed to explore rest of the endpoint setup.

How often are you hitting the endpoints? I think we should limit how often we hit them, I don't want CyVerse to have any reason to start blocking more IP addresses. I think we would only to hit the endpoints every ~10 minutes.

I can set the duration of hit. I too felt that hitting the endpoints continuously is not a good idea.

Nowlan Freese Please review working on the below link.
http://ec2-3-230-218-102.compute-1.amazonaws.com:3000/d/O-Y6m-IGk/igb-endpoint-monitoring?orgId=1&from=now-30m&to=now&var-Endpoint=https:%2F%2Fde.cyverse.org%2Fterrain

Is there a plan to deploy this yet?

I'm wondering how we will get this moved from Chester's account to the lab infrastructure?

I think it may be possible for Chester to simply transfer ownership of the EC2 to the Loraine Lab account. Is that right?

Re-assigning to Chester Dias for his comment.

cc: Nowlan Freese

I could do the whole setup on an EC2 in a Loraine lab account.
[~aloraine] Just need the below EC2 specifications
Instance Type: t2.micro
AMI to use: ami-0a313d6098716f372

I would need a domain and certificate files.

I could just do simple sftp transfers and the boards are already attached in this Jira that should be easy. I have done this kind of migration before.

Thanks!

New EC2 set up and available for your use. Please see Loraine Lab EC2 console. VM Name is "Grafana Host".

54.147.67.176 is now associated with grafana.bioviz.org in DNS. changes should take effect in a few hours.

https://grafana.bioviz.org/login is active. I will change the admin password and send it to you for creating more users
[~aloraine] We will need to use some email ID for the email configuration setup.
Please provide me with the user and password for this account
igbjira@gmail.com

Sent via separate email.

[~aloraine] I have put in the email configuration but it still needs the following change to be done to the email account
Login to : https://myaccount.google.com/u/0/security
Turn on Less secure app access

Turn off MFA if it is enabled.

I tried loging in to the account but it is asking me for an OTP

I just now logged in to check. Both of these settings are already done.
Note that the current Jira site is already using this email account to send mail. (Check the sender field on a message received from Jira.)
However, I see a security report in the Google console letting me know there was an attempt to sign in using the password from Virginia, which is probably the Grafana EC2 attempting to send email. I clicked an option to let Google know that this access is authorized.

Email setup is complete. I am still trying to get the notifications emails to fire properly.

Monitoring has been setup

Chester Dias - Please work with Nowlan Freese to configure the monitoring and show him how to use the site. Also do please walk him through the process of logging in and administering the site.

Also, please make sure the site setup is documented well enough so that we can back it up and restore it if needed.

will do that

Nowlan Freese Please let me know the status of the points we discussed during the meeting.

[~aloraine] - Chester and I had a couple of questions:

Should we configure the database to only store the last ~2 weeks of data?

• AL - this sounds fine. Probably you do not need that much. Mainly you just need to make sure that if the API changes you will know about it.

We are not sure how much data is generated every 24 hours, but there are around 30 endpoints generating data every 10 minutes.

• AL - every ten minutes is too much. Once an hour or even once a day is probably fine.

What do we need to do to configure the EC2 to not be shut down each night?

• AL - Change "Stack" tag to "Prod"

Since these questions may expand the scope of this ticket we may need to close this ticket and create additional tickets.

I've created a brief document with the notes I took from Chester's demo: https://docs.google.com/document/d/1U6VMjM7FsxGGCsYvaCcsWSBrnhQQwwun2GA-iLC8NWE/edit?usp=sharing

Please create a Jira ticket to automate the startup process.

Currently the steps are manual as per the documentation above:

In the event that the EC2 is turned off, the various services need to be manually turned back on in the correct order for the dashboard to work correctly:

cd /home/ubuntu/influxdb-1.8.2-1/usr/bin
nohup ./influxd -config ../../etc/influxdb/influxdb.conf &
cd /home/ubuntu/telegraf-1.15.3/usr/bin
nohup ./telegraf -config ../../etc/telegraf/telegraf.conf &
cd /home/ubuntu/grafana-7.2.0/bin
nohup ./grafana-server &

Created IGBF-2570.