Changes are implemented. See Commit

Commit looks good to me. Ready for pull request.

Note that I have not tested the code, but I think we can go ahead and push it since the converter is still unreleased and the only changes are to the UI.

PR Submitted. https://bitbucket.org/KarthikRavee91/hub-facade/pull-requests/1

Looks like the merge target is Karthik's fork. Change the PR to target the team repo and I can merge it then.

New PR Submitted with the correct target. https://bitbucket.org/lorainelab/hub-facade/pull-requests/19

I have updated translate.bioviz.org with the new code and there is an error. Please see attached images:

• old-ui.png
• new-ui.png (broken)

Testing translate.bioviz.org with IGB 9.1.10 on Mac.

Adding the public trackhubs to IGB from https://www.bioviz.org/public-trackhubs.html works correctly as well as the Open in IGB link. The hubs were correctly added to IGB as new Quickloads and I was able to load and view the data. This indicates that the API endpoint for translate.bioviz.org is still working correctly.

The UI changes to translate.bioviz.org appear correctly. The trackhub converter UI for translate.bioviz.org has been removed as it was an older duplicate of what we host at https://www.bioviz.org/trackhub-converter.html. As translate.bioviz.org is the API endpoint the UI has been replaced with a test of its endpoint using the UCSC provided example hub. The new-ui.png image is correct in that it shows that the endpoint is functioning correctly.

Closing ticket.

I am very sorry my involvement caused confusion for everybody. I ought to have better explained how the goal of this ticket changed as work progressed on it.

Main idea is:

• The UI available at https://translate.bioviz.org needs to exist and make sense to enable interactive testing by developers, testers, and users.

It is needed for effective stand-alone deployment for users / developers who do not have ability to set up their own instance of a secondary Web site such as the bioviz.org Web site. This is why it is needed - as a convenience for people not wanting to deal with the much larger and more complex code base of the larger Web site (https://bioviz.org).

Notes on improving the user interface are in a related ticket's comment here.

The ticket description has been changed, here is the old description:

Situation: translate.bioviz.org is the domain for the API that is used to convert UCSC Track Hubs to IGB Quickloads. For example, the following URL is queried by IGB to obtain the IGB Quickload contents.txt translated from the Track Hubs hub.txt file: https://translate.bioviz.org/api/?hubUrl=https://raw.githubusercontent.com/mhalushka/miROme/main/UCSC/hub.txt&filePath=/

The user interface for converting or viewing public Track Hubs is provided by bioviz.org/trackhub-converter.html and bioviz.org/public-trackhubs.html. However, if a user does navigate to translate.bioviz.org they will see an older version of the trackhub converter user interface. While there is nothing wrong with the interface, it is unnecessary as bioviz.org/trackhub-converter.html already provides the same user interface and is easier to develop and maintain.

Task: Remove the trackhub converter user interface from translate.bioviz.org. This will make the repository smaller and easier to understand. If a user does navigate to translate.bioviz.org we could provide a simple text output that informs the user the API is currently working. This is similar to what CyVerse does - https://de.cyverse.org/terrain

The UI for the translate.bioviz.org was changed back to latest version of manual converter. However the navbar menu needs to be pulled from bioviz.org same as Appstore. Refer to IGBF-2655 code

Pull navigation bar from bioviz.org implemented in hub-facade. See commit

PR Submitted https://bitbucket.org/lorainelab/hub-facade/pull-requests/20

PR is merged. New code deployed to https://translate.bioviz.org. Ready for testing.

Re-ran playbooks to see if error mentioned in scrum goes away.
attn: Nowlan Freese

Getting a 500 Internal Server Error:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.
Apache/2.4.29 (Ubuntu) Server at translate.bioviz.org Port 443

The error is caused because the application pulls the centralized menu from bioviz.org and pastes the code to local project files, menu.css and menu.html. However, the path to menu.css and menu.html is currently specified as relative to the project - it has to be absolute. In appstore, the path is built by appending the project path and the relative path of the files. However, this did not work in the local system in development in windows because the format of the filepath is different from linux ('\ \' instead of '/').

See Commit with changes for the above comment: https://bitbucket.org/KarthikRavee91/hub-facade/commits/177598d058220030489a8350a84f452abd420768
PR Submitted: https://bitbucket.org/lorainelab/hub-facade/pull-requests/21

Although the above Internal Server Error has been resolved a new error has been observed by Nowlan Freese.

 mod_wsgi (pid=3428): Target WSGI script '/var/www/vhosts/translate.bioviz.org/igb_trackhub/igb_trackhub/wsgi.py' cannot be loaded as Python module.
 mod_wsgi (pid=3428): Exception occurred processing WSGI script '/var/www/vhosts/translate.bioviz.org/igb_trackhub/igb_trackhub/wsgi.py'.
 Traceback (most recent call last):
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/igb_trackhub/wsgi.py", line 16, in <module>
     application = get_wsgi_application()
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/venv/lib/python3.6/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
     django.setup(set_prefix=False)
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/venv/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
     apps.populate(settings.INSTALLED_APPS)
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/venv/lib/python3.6/site-packages/django/apps/registry.py", line 83, in populate
     raise RuntimeError("populate() isn't reentrant")
 RuntimeError: populate() isn't reentrant
 mod_wsgi (pid=3428): Target WSGI script '/var/www/vhosts/translate.bioviz.org/igb_trackhub/igb_trackhub/wsgi.py' cannot be loaded as Python module.
 mod_wsgi (pid=3428): Exception occurred processing WSGI script '/var/www/vhosts/translate.bioviz.org/igb_trackhub/igb_trackhub/wsgi.py'.
 Traceback (most recent call last):
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/igb_trackhub/wsgi.py", line 16, in <module>
     application = get_wsgi_application()
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/venv/lib/python3.6/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
     django.setup(set_prefix=False)
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/venv/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
     apps.populate(settings.INSTALLED_APPS)
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/venv/lib/python3.6/site-packages/django/apps/registry.py", line 83, in populate
     raise RuntimeError("populate() isn't reentrant")
 RuntimeError: populate() isn't reentrant
 mod_wsgi (pid=3428): Target WSGI script '/var/www/vhosts/translate.bioviz.org/igb_trackhub/igb_trackhub/wsgi.py' cannot be loaded as Python module.
 mod_wsgi (pid=3428): Exception occurred processing WSGI script '/var/www/vhosts/translate.bioviz.org/igb_trackhub/igb_trackhub/wsgi.py'.
 Traceback (most recent call last):
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/igb_trackhub/wsgi.py", line 16, in <module>
     application = get_wsgi_application()
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/venv/lib/python3.6/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
     django.setup(set_prefix=False)
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/venv/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
     apps.populate(settings.INSTALLED_APPS)
   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/venv/lib/python3.6/site-packages/django/apps/registry.py", line 83, in populate
     raise RuntimeError("populate() isn't reentrant")
 RuntimeError: populate() isn't reentrant

This could be caused due to version of django that is being used. In appstore, the version is 3.1 whereas hub-facade uses 4.0.9. It could possible be mysqlclient issue as well,unclear on which it is.

I was able to run the playbook for translate.bioviz.org. When Karthik has new changes I can point it at his branch and run the playbook to test the changes.

Pull request for adding the pubkey role to setup.yml: https://bitbucket.org/lorainelab/hub-facade/pull-requests/23

These two codes are problematic because the python is not able to access the files menu.css and menu.html

css_file = open(path.join(current_path,'../ui/static/css/menu.css'), 'w')
html_file = open(path.join(current_path,'../ui/templates/menu.html'), 'w')

To resolve this from what I understand ownership of the files should be changed from root.

chown ubuntu:admin /var/www/vhosts/translate.bioviz.org/igb_trackhub/ui/static/css/menu.css
chmod 755 /var/www/vhosts/translate.bioviz.org/igb_trackhub/ui/static/css/menu.css

chown ubuntu:admin /var/www/vhosts/translate.bioviz.org/igb_trackhub/ui/templates/menu.html
chmod 755 /var/www/vhosts/translate.bioviz.org/igb_trackhub/ui/templates/menu.html

However, I do not have permission for this change in ownership. I commented the problematic code and it works and new changes in the centralised menu is available but further changes will not which defeats the purpose.

One of the appstore commits that seems like it implements borring the menu.html from the bioviz codebase: https://bitbucket.org/lorainelab/appstore/commits/91d7d20ff275b7077e6d6989f862d98b184497ea

Some menu items in the centralized menu was pointing to the local project(translate.bioviz.org) where the files do not exist. It should point to bioviz.org instead. Changes have been implemented. See Commit

PR Submitted: https://bitbucket.org/lorainelab/hub-facade/pull-requests/25
Included:

• Remove unused code
• Fix menu items links pointing to local project rather than bioviz.org
• Add toast messages for Add to IGB button

PR: https://bitbucket.org/lorainelab/hub-facade/pull-requests/28

Karthik found that the issue resulting in the 500 error was due to the Apache www-data group not having permission to access the menu.css and menu.html files. App store handled this by making Ubuntu the owner of the www-data group. We migrated this code to be used with hub-facade.

Note: The changes we made to the playbook and code appear to be working. However, we had made manual modifications to the server while testing. I think it would be a good idea to try running the playbook on a new ec2 to see if everything is working correctly.

Documenting debug process for future use:
The runtime error shown below, raised in python registry, is not very helpful in finding where the error is happening. This is issue has been fixed in newer versions of python.

   File "/var/www/vhosts/translate.bioviz.org/igb_trackhub/venv/lib/python3.6/site-packages/django/apps/registry.py", line 83, in populate
     raise RuntimeError("populate() isn't reentrant")
 RuntimeError: populate() isn't reentrant

However, we are using an older python version so this error message is subdued by clearing this error so the underlying problem is revealed, which is shown below.

PermissionError: [Errno 13] Permission denied: '/var/www/vhosts/translate.bioviz.org/igb_trackhub/api/../ui/static/css/menu.css'

Couple questions:

• At what point during the operation of the application do the files "menu.css" and "menu.html" get retrieved from the external bitbucket resources? Please indicate code where this is done, if possible.
• Under what user id is the application itself running? (maybe under the same uuid as the apache web server process ?)

For security reasons the menu.css and menu.html files are retrieved from bioviz.org and copied to a separate folders, named common-menu.
PR Submitted: https://bitbucket.org/lorainelab/hub-facade/pull-requests/27

PR is merged. Ready for deployment to translate.bioviz.org.
attn: Nowlan Freese

Also, PR for ansible-playbook change is now merged.

I have run the playbook and tested on Firefox.
The navbar links work correctly.
No errors observed in the console.
I was able to convert the example trackhub url and add it to IGB.

Closing ticket.