Details
-
Type:
Bug
-
Status: Closed (View Workflow)
-
Priority:
Blocker
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:None
-
Story Points:2
-
Epic Link:
-
Sprint:Fall 2 2021 Aug 30 - Sep10, Fall 3 2021 Sep 13 - Sep 24
Description
Tuesday morning and two weeks ago, when I attempted to use BioViz Connect, I logged in successfully, but the Web site showed only a blank page.
For this task,
- determine the source of the error and describe it in the comments below
- fix the error so that it does not occur again
- modify the code to show a useful and friendly notice to the user to let them know that the site is unavailable and will be back on-line soon
Attachments
Issue Links
- relates to
-
IGBF-2950 Investigate how to make the site more robust to back end CyVerse errors
-
- Closed
-
Activity
Reviewing error log:
bvctest3.bioviz.org ubuntu $ cat /var/log/apache2/error.log
[Tue Sep 07 17:46:39.623945 2021] [mpm_event:notice] [pid 943:tid 139901364321216] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations [Tue Sep 07 17:46:39.670418 2021] [core:notice] [pid 943:tid 139901364321216] AH00094: Command line: '/usr/sbin/apache2' [Tue Sep 07 17:53:05.112177 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 36.68.9.26:22407] Invalid HTTP_HOST header: '3.216.236.83'. You may need to add '3.216.236.83' to ALLOWED_HOSTS. [Tue Sep 07 17:53:05.168826 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 36.68.9.26:22407] Bad Request: /.env [Tue Sep 07 17:54:34.724446 2021] [wsgi:error] [pid 944:tid 139901113681664] [remote 75.90.51.92:54542] Not Found: /favicon.ico [Tue Sep 07 17:54:39.359194 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] Internal Server Error: /getSessionId/ [Tue Sep 07 17:54:39.359227 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] Traceback (most recent call last): [Tue Sep 07 17:54:39.359232 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] File "/var/www/vhosts/bioviz-connect/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner [Tue Sep 07 17:54:39.359236 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] response = get_response(request) [Tue Sep 07 17:54:39.359241 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] File "/var/www/vhosts/bioviz-connect/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 179, in _get_response [Tue Sep 07 17:54:39.359246 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] response = wrapped_callback(request, *callback_args, **callback_kwargs) [Tue Sep 07 17:54:39.359250 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] File "/var/www/vhosts/bioviz-connect/bioviz_connect/views.py", line 124, in getSessionId [Tue Sep 07 17:54:39.359255 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] user_name = getUserNamefromSessionData(redis_sessiondata) [Tue Sep 07 17:54:39.359259 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] File "/var/www/vhosts/bioviz-connect/bioviz_connect/terrain/getSessionUserData.py", line 16, in getUserNamefromSessionData [Tue Sep 07 17:54:39.359264 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] response = ResponseParser.parse_getUserDetails(json_data[1]) [Tue Sep 07 17:54:39.359268 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] File "/var/www/vhosts/bioviz-connect/bioviz_connect/terrain_model/ResponseParser.py", line 388, in parse_getUserDetails [Tue Sep 07 17:54:39.359272 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] response = ResponseParser.parse_userDetailresponse(api_response) [Tue Sep 07 17:54:39.359276 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] File "/var/www/vhosts/bioviz-connect/bioviz_connect/terrain_model/ResponseParser.py", line 62, in parse_userDetailresponse [Tue Sep 07 17:54:39.359280 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] for root in response['roots']: [Tue Sep 07 17:54:39.359286 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542] KeyError: 'roots' [Tue Sep 07 17:54:39.359308 2021] [wsgi:error] [pid 944:tid 139901231179520] [remote 75.90.51.92:54542]
Restarted apache on production host:
[Tue Sep 07 18:27:18.722365 2021] [mpm_event:notice] [pid 19659:tid 140200438442944] AH00491: caught SIGTERM, shutting down [Tue Sep 07 18:27:18.829484 2021] [mpm_event:notice] [pid 31519:tid 140249383418816] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations [Tue Sep 07 18:27:18.829575 2021] [core:notice] [pid 31519:tid 140249383418816] AH00094: Command line: '/usr/sbin/apache2' [Tue Sep 07 18:28:53.433823 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54752] Bad Request: /getRootPathUser/ [Tue Sep 07 18:29:48.721765 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] Internal Server Error: /getSessionId/ [Tue Sep 07 18:29:48.721937 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] Traceback (most recent call last): [Tue Sep 07 18:29:48.722019 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] File "/var/www/vhosts/bioviz-connect/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner [Tue Sep 07 18:29:48.722088 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] response = get_response(request) [Tue Sep 07 18:29:48.722155 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] File "/var/www/vhosts/bioviz-connect/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 179, in _get_response [Tue Sep 07 18:29:48.722237 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] response = wrapped_callback(request, *callback_args, **callback_kwargs) [Tue Sep 07 18:29:48.722299 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] File "/var/www/vhosts/bioviz-connect/bioviz_connect/views.py", line 124, in getSessionId [Tue Sep 07 18:29:48.722369 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] user_name = getUserNamefromSessionData(redis_sessiondata) [Tue Sep 07 18:29:48.722432 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] File "/var/www/vhosts/bioviz-connect/bioviz_connect/terrain/getSessionUserData.py", line 16, in getUserNamefromSessionData [Tue Sep 07 18:29:48.722497 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] response = ResponseParser.parse_getUserDetails(json_data[1]) [Tue Sep 07 18:29:48.722549 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] File "/var/www/vhosts/bioviz-connect/bioviz_connect/terrain_model/ResponseParser.py", line 388, in parse_getUserDetails [Tue Sep 07 18:29:48.722612 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] response = ResponseParser.parse_userDetailresponse(api_response) [Tue Sep 07 18:29:48.722666 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] File "/var/www/vhosts/bioviz-connect/bioviz_connect/terrain_model/ResponseParser.py", line 62, in parse_userDetailresponse [Tue Sep 07 18:29:48.722710 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] for root in response['roots']: [Tue Sep 07 18:29:48.722753 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787] KeyError: 'roots' [Tue Sep 07 18:29:48.722802 2021] [wsgi:error] [pid 31522:tid 140249132779264] [remote 75.90.51.92:54787]
Our code hardcodes the name "iplant" - see "getSessionUserData.py":
def getUserNamefromSessionData(json_data):
base_prefixhomepath = '/iplant/home/'
response = ResponseParser.parse_getUserDetails(json_data[1])
homepath = response.base_paths.user_home_path
return homepath.split(base_prefixhomepath)[1]
Request for Nowlan Freese:
Can you check to see if CyVerse changed the "base_prefixhomepath" ? I bet they changed it from "iplant/home" to something else.
This post to the UCSC genome browser list appeared today. It might be related somehow:
Hi there, I used to be able to successfully view bam files through cyverse.org. However, recently, the cyverse link did not seem to work: the UCSC genome browser showed only long and short labels for the file, but no reads were deplayed, in contrast, when I replaced with a different server link that hosting the same bam file, it displayed well. I wonder if you can help me figure out what I can do to continue to use cyverse? Here is an example that it used to work and still works https://data.cyverse.org/dav-anon/iplant/home/jingliu352/analyses/mm10/5umchir_cas9_gfpsgrna.bam here is an example that does not work. I uploaded the file through Cyberduck. https://data.cyverse.org/dav-anon/iplant/home/jingliu352/analyses/hg38/SRR10303461_M74.bam I look forward to hearing from you soon, thanks! Jing Liu, Ph.D. University of Southern California
CyVerse and related resources are undergoing maintenance, future dates from their webpage are:
September 7, 2021
October 5, 2021
November 2, 2021
December 7, 2021
Reply from UCSC regarding unavailable file question in preceding comment:
Brian Lee <brianlee@soe.ucsc.edu> to: "Liu, Jing" <Jing.Liu@med.usc.edu> cc: "genome@soe.ucsc.edu" <genome@soe.ucsc.edu> date: Sep 7, 2021, 3:54 PM subject: [EXTERNAL] Re: [genome] RE: trouble viewing bam files mailing list: genome@soe.ucsc.edu Filter messages from this mailing list mailed-by: soe.ucsc.edu Dear Jing, Thank you for using the UCSC Genome Browser and your question about having trouble viewing bam files. It looks like at this moment the CyVerse site is having maintenance: https://cyverse.org/maintenance They have started a cycle of using the first Tuesday of every month from 7 am - 7 pm Pacific Time to perform work. These dates will be impacted: September 7, 2021 October 5, 2021 November 2, 2021 December 7, 2021 However, just in case after they are back online you are still having the issue, be sure that you have made the files for both the file.bam and the file.bam.bai public on the CyVerse site. In the past, I believe CyVerse had automatic software to make any file.bam and its related file.bam.bai public, and I believe it may be different now where you may have to select both files and mark them as public manually. You can see some illustrated steps of their updated system here, http://genome.ucsc.edu/goldenPath/help/hgTrackHubHelp.html#Hosting, where you may also want to read about inquiring with your institution about hosting data. Thank you again for your inquiry and for using the UCSC Genome Browser. If you have any further public questions, please reply to genome@soe.ucsc.edu. All messages sent to that address are archived on a publicly accessible forum. If your question includes sensitive data, you may send it instead to genome-www@soe.ucsc.edu. All the best,
We need to notice these problems immediately - before users do, anyway.
Possible solutions:
- Write an app that notifies site administrators when an error occurs
- Configure Apache to report an error to the user when there's an internal server error, detected as text "[wsgi:error]" appearing in the log, first making sure that this error message only appears in blocker situations
Describe how to test any changes made to the code. Come up with a strategy that will let the testers simulate CyVerse API failing.
As discussed yesterday during the meeting, the issue was caused because the Cyverse server was down and the errors were improperly handled, specifically in ResponseParser.py in the terrain_model folder.
Possible Solution: Custom error handling classes built for API calls made in the application. How the error classes should be designed and placed in the code base is yet to be determined. The error handling should avoid code smells, useful information should be sent to the UI, and should be class-specific for easier debugging. Also, a testing protocol should be designed as well for checking all the errors before deployment.
Loraine: Needs to create a ticket describing the next phase of making the site more robust to back end CyVerse errors.
Started bvctest3.bioviz.org to review logs and diagnose problem.