Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-3534

Update deployment strategy to distribute JDK 21 IGB and early access installers

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      Fall 8, Spring 1, Spring 2, Spring 3, Spring 4, Spring 5

      Description

      For this task:

      • Update deployment code to deploy new versions of IGB better
      • Distribute new IGB installers on bioviz.org "early access" page.

        Attachments

          Issue Links

            Activity

            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Update:

            Show
            ann.loraine Ann Loraine added a comment - - edited Update: Working on https://bitbucket.org/lorainelab/bioviz . Changing main branch to "main" instead of "master." Doing early release updates first. Testing using target host bioviztest3.bioviz.org.
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Update:

            Show
            ann.loraine Ann Loraine added a comment - - edited Update: Making new version of playbooks under branch name "ubuntu" at https://bitbucket.org/lorainelab/bioviz-playbooks/src/ubuntu/ Made a new host (ubuntu OS) to serve content at http://biovizubuntutest.bioviz.org with IP address 44.221.242.103. Registered domain and IP with Network Solutions. Configuring the new host so that we can easily host multiple sites on the same VM (using "sites-enabled" scheme ubuntu has, which I like)
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Update:

            • Making changes to bitbucket pipeline configuration YML file: (1) removing reference to master branch (2) keeping original names of branch installers except replacing version with branch name
            • Pushed new branch IGBF-3534 to my IGB fork https://bitbucket.org/aloraine/integrated-genome-browser for testing
            • Checked that the new branch can build and produce the expected installer files using custom pipeline "build-installers-for-dev" and branch pipeline "release-*".

            The main difference between these two pipelines is that "release" inserts the IGB version in the file name while the "for-dev" pipeline inserts the branch name. Also, the "release" pipeline copies over the updates annots.xml file and a couple other things.

            Show
            ann.loraine Ann Loraine added a comment - - edited Update: Making changes to bitbucket pipeline configuration YML file: (1) removing reference to master branch (2) keeping original names of branch installers except replacing version with branch name Pushed new branch IGBF-3534 to my IGB fork https://bitbucket.org/aloraine/integrated-genome-browser for testing Checked that the new branch can build and produce the expected installer files using custom pipeline "build-installers-for-dev" and branch pipeline "release-*". The main difference between these two pipelines is that "release" inserts the IGB version in the file name while the "for-dev" pipeline inserts the branch name. Also, the "release" pipeline copies over the updates annots.xml file and a couple other things.
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Updated all the playbooks.

            To test, compare with bioviz.org:

            • Check that the link to galaxy can still work.
            • Check that the "bar" link can still work.
            • Check that everything under the "tools" menu can still work.
            • Click on all the links and check that they still work.
            • Check that download links work.
            Show
            ann.loraine Ann Loraine added a comment - - edited Updated all the playbooks. To test, compare with bioviz.org: Check that the link to galaxy can still work. Check that the "bar" link can still work. Check that everything under the "tools" menu can still work. Click on all the links and check that they still work. Check that download links work.
            Hide
            nfreese Nowlan Freese added a comment -

            Testing http://biovizubuntutest.bioviz.org/ on Mac with IGB 10.0.0 (from early access).

            • Links work correctly.
            • Download page shows button for Mac and clicking the button downloads the IGB_macos_current.dmg installer.
            • Early Access page shows button for Mac and clicking the button downloads the IGB-macos-universal-main-JDK21-with-javafx-no-theme.dmg installer (which works).
            • Galaxy works (example link).
            • Tools > View IGB Genomes is working correctly.
            • Tools > View UCSC Trackhubs is working correctly.

            Not working - BAR link fails with the following error:

            Failed to load resource: the server responded with a status of 404 (Not Found) - https://biovizubuntutest.bioviz.org/cgi-bin/geneIdLookup.py?gene_id=AT2G24270
            Show
            nfreese Nowlan Freese added a comment - Testing http://biovizubuntutest.bioviz.org/ on Mac with IGB 10.0.0 (from early access). Links work correctly. Download page shows button for Mac and clicking the button downloads the IGB_macos_current.dmg installer. Early Access page shows button for Mac and clicking the button downloads the IGB-macos-universal-main-JDK21-with-javafx-no-theme.dmg installer (which works). Galaxy works (example link ). Tools > View IGB Genomes is working correctly. Tools > View UCSC Trackhubs is working correctly. Not working - BAR link fails with the following error: Failed to load resource: the server responded with a status of 404 (Not Found) - https: //biovizubuntutest.bioviz.org/cgi-bin/geneIdLookup.py?gene_id=AT2G24270
            Hide
            ann.loraine Ann Loraine added a comment -

            Implementing changes to correct the above error.
            Problem was due to my forgetting to configure and enable CGI (common gateway interface) scripts to run.
            Looked up convention for deploying CGI script on an Ubuntu host.
            By default, apache2 on Ubuntu is configured to run cgi scripts present in directory /usr/lib/cgi-bin.
            The default configuration seems to allow links from the cloned repository's copy of the geneIdLookup.py script to a location in the default cgi-bin location of /usr/lib/cgi-bin.
            I decided to add a new task to the "clone" role to create this link to the central cgi-bin location.
            Running the playbook now for testing.

            Show
            ann.loraine Ann Loraine added a comment - Implementing changes to correct the above error. Problem was due to my forgetting to configure and enable CGI (common gateway interface) scripts to run. Looked up convention for deploying CGI script on an Ubuntu host. By default, apache2 on Ubuntu is configured to run cgi scripts present in directory /usr/lib/cgi-bin. The default configuration seems to allow links from the cloned repository's copy of the geneIdLookup.py script to a location in the default cgi-bin location of /usr/lib/cgi-bin. I decided to add a new task to the "clone" role to create this link to the central cgi-bin location. Running the playbook now for testing.
            Hide
            ann.loraine Ann Loraine added a comment -

            For our reference, here is the "out of the box" cgi configuration:

            biovizubuntutest.bioviz.org ubuntu $ pwd
            /etc/apache2/conf-enabled
            biovizubuntutest.bioviz.org ubuntu $ more serve-cgi-bin.conf 
            <IfModule mod_alias.c>
            	<IfModule mod_cgi.c>
            		Define ENABLE_USR_LIB_CGI_BIN
            	</IfModule>
            
            	<IfModule mod_cgid.c>
            		Define ENABLE_USR_LIB_CGI_BIN
            	</IfModule>
            
            	<IfDefine ENABLE_USR_LIB_CGI_BIN>
            		ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
            		<Directory "/usr/lib/cgi-bin">
            			AllowOverride None
            			Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
            			Require all granted
            		</Directory>
            	</IfDefine>
            </IfModule>
            
            Show
            ann.loraine Ann Loraine added a comment - For our reference, here is the "out of the box" cgi configuration: biovizubuntutest.bioviz.org ubuntu $ pwd /etc/apache2/conf-enabled biovizubuntutest.bioviz.org ubuntu $ more serve-cgi-bin.conf <IfModule mod_alias.c> <IfModule mod_cgi.c> Define ENABLE_USR_LIB_CGI_BIN </IfModule> <IfModule mod_cgid.c> Define ENABLE_USR_LIB_CGI_BIN </IfModule> <IfDefine ENABLE_USR_LIB_CGI_BIN> ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ <Directory "/usr/lib/cgi-bin" > AllowOverride None Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch Require all granted </Directory> </IfDefine> </IfModule>
            Hide
            ann.loraine Ann Loraine added a comment -

            Looks like the changes that I made fixed the problem.
            Ready for testing.

            Show
            ann.loraine Ann Loraine added a comment - Looks like the changes that I made fixed the problem. Ready for testing.
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Update:

            Wrote new plays / playbooks to deploy maven.bioviz.org content on the same host as the main web site bioviz.org.
            I tested it on biovizubuntutest.bioviz.org and it looked fine.

            However, when I re-deployed as www.bioviz.org and maven.bioviz.org, there was a problem. To test it, I added these lines to my /etc/hosts file to "trick" my computer in using the newly deployed ubuntu-based bioviz.org site. (I did this for testing.)

            44.221.227.251 www.bioviz.org
            44.221.227.251 bioviz.org
            

            The problem is that when I try to visit bioviz.org or www.bioviz.org, I get redirected to the maven site instead of bioviz main site.

            Show
            ann.loraine Ann Loraine added a comment - - edited Update: Wrote new plays / playbooks to deploy maven.bioviz.org content on the same host as the main web site bioviz.org. I tested it on biovizubuntutest.bioviz.org and it looked fine. However, when I re-deployed as www.bioviz.org and maven.bioviz.org, there was a problem. To test it, I added these lines to my /etc/hosts file to "trick" my computer in using the newly deployed ubuntu-based bioviz.org site. (I did this for testing.) 44.221.227.251 www.bioviz.org 44.221.227.251 bioviz.org The problem is that when I try to visit bioviz.org or www.bioviz.org, I get redirected to the maven site instead of bioviz main site.
            Hide
            nfreese Nowlan Freese added a comment - - edited

            Attempting to download early access installers from https://www.bioviz.org/earlyAccessDownload.html

            Mac - worked, downloaded IGB-macos-universal-main-JDK21-with-javafx-no-theme.dmg
            Linux - did not work, saw a 404 when I clicked the early access Linux button - bioviz.org/igb/releases/early_access/IGB-linux-main-JDK21-with-javafx-no-theme_amd64.sh
            Windows - did not work, saw a 404 when I clicked the early access Windows button - bioviz.org/igb/releases/early_access/IGB-windows-main-JDK21-with-javafx-no-theme_amd64.exe

            On the https://www.bioviz.org/otherDownloads.html page there is the option for Windows 32 bit, which I don't think we support. Clicking the button results in a 404.

            Show
            nfreese Nowlan Freese added a comment - - edited Attempting to download early access installers from https://www.bioviz.org/earlyAccessDownload.html Mac - worked, downloaded IGB-macos-universal-main-JDK21-with-javafx-no-theme.dmg Linux - did not work, saw a 404 when I clicked the early access Linux button - bioviz.org/igb/releases/early_access/IGB-linux-main-JDK21-with-javafx-no-theme_amd64.sh Windows - did not work, saw a 404 when I clicked the early access Windows button - bioviz.org/igb/releases/early_access/IGB-windows-main-JDK21-with-javafx-no-theme_amd64.exe On the https://www.bioviz.org/otherDownloads.html page there is the option for Windows 32 bit, which I don't think we support. Clicking the button results in a 404.
            Hide
            ann.loraine Ann Loraine added a comment -

            Update:

            • Removed Windows 32 bit link
            • Modified links so that early access buttons work correctly
            Show
            ann.loraine Ann Loraine added a comment - Update: Removed Windows 32 bit link Modified links so that early access buttons work correctly
            Hide
            nfreese Nowlan Freese added a comment - - edited

            Testing:

            • Windows 32 bit link has been removed.
            • The Windows and LInux early access buttons are pointed at the correct files.

            Note: I tested this on Mac by modifying my user agent to test Windows/Linux. The Windows button worked to download the exe. The Linux button when clicked attempted to open the .sh file in the browser as a text file for both the downloads page and early access. I am unsure if this is a result of being on Mac. I will double-check with a Linux machine to make sure that the .sh file is being downloaded properly.

            UPDATE: Tested the Linux download button on the downloads page and the early access page, they both worked correctly.

            Show
            nfreese Nowlan Freese added a comment - - edited Testing: Windows 32 bit link has been removed. The Windows and LInux early access buttons are pointed at the correct files. Note: I tested this on Mac by modifying my user agent to test Windows/Linux. The Windows button worked to download the exe. The Linux button when clicked attempted to open the .sh file in the browser as a text file for both the downloads page and early access. I am unsure if this is a result of being on Mac. I will double-check with a Linux machine to make sure that the .sh file is being downloaded properly. UPDATE: Tested the Linux download button on the downloads page and the early access page, they both worked correctly.
            Hide
            nfreese Nowlan Freese added a comment -

            Testing BAR link:

            The BAR link is currently not working as SciDas is down. As we are navigating away from using SciDas (see IGBF-3614), I have created a pull request updating the BioViz code to use http://bioviz.org/quickload/bar/ instead.

            Pull request: https://bitbucket.org/lorainelab/bioviz/pull-requests/126

            Show
            nfreese Nowlan Freese added a comment - Testing BAR link : The BAR link is currently not working as SciDas is down. As we are navigating away from using SciDas (see IGBF-3614), I have created a pull request updating the BioViz code to use http://bioviz.org/quickload/bar/ instead. Pull request: https://bitbucket.org/lorainelab/bioviz/pull-requests/126
            Hide
            ann.loraine Ann Loraine added a comment -

            I have merged the PR.

            I have a suggestion:

            The BAR quickload site consists of metadata files only. How about if we point IGB to this location instead: https://bioviz.org/quickload/bar/

            I'm not exactly sure why we are using scidas to deploy the bar quickload site. Could it be something to do with http versus https?

            I am moving this back to "To-Do" for Nowlan Freese to weigh in on this suggestion.

            Show
            ann.loraine Ann Loraine added a comment - I have merged the PR. I have a suggestion: The BAR quickload site consists of metadata files only. How about if we point IGB to this location instead: https://bioviz.org/quickload/bar/ I'm not exactly sure why we are using scidas to deploy the bar quickload site. Could it be something to do with http versus https? I am moving this back to "To-Do" for Nowlan Freese to weigh in on this suggestion.
            Hide
            nfreese Nowlan Freese added a comment -

            I found an issue with how the server_url is being used with BAR. The current server_url being served from bioviz.org/bar.html is http://lorainelab-quickload.scidas.org/bar/

            This is problematic as BookmarkUnibrowControlServlet is seeing the server_url and attempting to match it with a data provider. From my testing, if the server_url and the quickload url do not match, it is causing an error and the data are not loading. For example, by changing the server_url to http://bioviz.org/quickload/bar/ we will break BAR data loading in IGB 9.1.10 as it will not match the data provider http://lorainelab-quickload.scidas.org/bar/. This is different than how Galaxy and BioViz Connect work, where the server_url is provided as "cyverse" or "galaxy". The server_url can also be left empty, but this would cause the data to load as a "local file" and would not have the formatting provided by the quickload site.

            My recommendation would be to roll back the above change to bioviz.org as well as to keep the primary url as http://lorainelab-quickload.scidas.org/bar

            Show
            nfreese Nowlan Freese added a comment - I found an issue with how the server_url is being used with BAR. The current server_url being served from bioviz.org/bar.html is http://lorainelab-quickload.scidas.org/bar/ This is problematic as BookmarkUnibrowControlServlet is seeing the server_url and attempting to match it with a data provider. From my testing, if the server_url and the quickload url do not match, it is causing an error and the data are not loading. For example, by changing the server_url to http://bioviz.org/quickload/bar/ we will break BAR data loading in IGB 9.1.10 as it will not match the data provider http://lorainelab-quickload.scidas.org/bar/ . This is different than how Galaxy and BioViz Connect work, where the server_url is provided as "cyverse" or "galaxy". The server_url can also be left empty, but this would cause the data to load as a "local file" and would not have the formatting provided by the quickload site. My recommendation would be to roll back the above change to bioviz.org as well as to keep the primary url as http://lorainelab-quickload.scidas.org/bar
            Hide
            ann.loraine Ann Loraine added a comment -

            Noted. I will make the change.

            Thank you for catching this Nowlan Freese!

            Show
            ann.loraine Ann Loraine added a comment - Noted. I will make the change. Thank you for catching this Nowlan Freese !
            Hide
            nfreese Nowlan Freese added a comment -

            I have reverted the commit back to using http://lorainelab-quickload.scidas.org/bar to align with igbDefaultPrefs.json

            Pull request: https://bitbucket.org/lorainelab/bioviz/pull-requests/127

            Show
            nfreese Nowlan Freese added a comment - I have reverted the commit back to using http://lorainelab-quickload.scidas.org/bar to align with igbDefaultPrefs.json Pull request: https://bitbucket.org/lorainelab/bioviz/pull-requests/127
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            PR is merged. Deployed new content to the VM which is hosting Bioviz.org.

            Show
            ann.loraine Ann Loraine added a comment - - edited PR is merged. Deployed new content to the VM which is hosting Bioviz.org.
            Hide
            ann.loraine Ann Loraine added a comment -

            Dr. Freese: Could you provide a link to instructions needed to test that the "bar" feature is working as expected?

            cc: Nowlan Freese

            Show
            ann.loraine Ann Loraine added a comment - Dr. Freese: Could you provide a link to instructions needed to test that the "bar" feature is working as expected? cc: Nowlan Freese
            Hide
            nfreese Nowlan Freese added a comment - - edited

            To test the BAR feature in IGB:

            1. Start IGB 10.0.0
            2. In a web browser, navigate to https://bar.utoronto.ca/eFP-Seq_Browser/?locus=AT2G24270&dataset=cgi-bin/data/bamdata_araport11.xml
            3. Look for the sample: E-MTAB-1668:24hCS-RNA
            4. Click Alignments in IGB
            5. A webpage should open (https://bioviz.org/bar.html)
            6. IGB should change to the Arabidopsis genome, move to the appropriate gene (AT2G24270) and load the selected data
            • Check that the track name is listed as: ERR274310 Aerial part of long-day-grown 4-leaf-stage seedling with mock (NaCl) treatment
            • Check that the foreground color of the track is green
            • Check that there are no errors in the IGB log
            Show
            nfreese Nowlan Freese added a comment - - edited To test the BAR feature in IGB: Start IGB 10.0.0 In a web browser, navigate to https://bar.utoronto.ca/eFP-Seq_Browser/?locus=AT2G24270&dataset=cgi-bin/data/bamdata_araport11.xml Look for the sample: E-MTAB-1668:24hCS-RNA Click Alignments in IGB A webpage should open ( https://bioviz.org/bar.html ) IGB should change to the Arabidopsis genome, move to the appropriate gene (AT2G24270) and load the selected data Check that the track name is listed as: ERR274310 Aerial part of long-day-grown 4-leaf-stage seedling with mock (NaCl) treatment Check that the foreground color of the track is green Check that there are no errors in the IGB log
            Hide
            nfreese Nowlan Freese added a comment - - edited

            Ann Loraine - The IGB bitbucket docker file lists the old JREs. I know we switched to using the new JRE (21.0.2) as part of IGBF-3588 but am unsure if we also need to update the docker file. Were these changes done locally and not pushed to the repo?

            Show
            nfreese Nowlan Freese added a comment - - edited Ann Loraine - The IGB bitbucket docker file lists the old JREs. I know we switched to using the new JRE (21.0.2) as part of IGBF-3588 but am unsure if we also need to update the docker file. Were these changes done locally and not pushed to the repo?
            Hide
            ann.loraine Ann Loraine added a comment -

            Need to commit the new Dockerfile to the git repo.

            Show
            ann.loraine Ann Loraine added a comment - Need to commit the new Dockerfile to the git repo.
            Hide
            nfreese Nowlan Freese added a comment - - edited

            Commit to update the docker file to use JRE 21.0.2: https://bitbucket.org/lorainelab/integrated-genome-browser-docker/commits/e1ee5b69243dc725cf8b6e2e3313940781a589bb

            I have also tested BAR using the IGB 10.0.0 release installer, it is working correctly.

            Closing ticket.

            Show
            nfreese Nowlan Freese added a comment - - edited Commit to update the docker file to use JRE 21.0.2: https://bitbucket.org/lorainelab/integrated-genome-browser-docker/commits/e1ee5b69243dc725cf8b6e2e3313940781a589bb I have also tested BAR using the IGB 10.0.0 release installer, it is working correctly. Closing ticket.

              People

              • Assignee:
                ann.loraine Ann Loraine
                Reporter:
                ann.loraine Ann Loraine
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: