Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-3898

Remove DAS data provider from igbDefaultPrefs.json

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 10.1.0
    • Labels:
      None

      Description

      Situation: In IGB 10.1.0 we are adding the UCSC REST API data provider to replace the UCSC DAS data provider. On the current IGB main branch we have set the DAS data provider to be disabled. If IGB 10.1.0 is installed on a new system, DAS is disabled. However, if a user goes from 10.0.1 to 10.1.0 the DAS data provider remains enabled. Having both DAS and UCSC REST enabled slows IGB down it requires additional API calls.

      Task: Remove the DAS data provider from the igbDefaultPrefs.json file. This will work to remove DAS from both new installs of IGB as well as updates from a previous version of IGB to the new version.

        Attachments

          Activity

          nfreese Nowlan Freese created issue -
          nfreese Nowlan Freese made changes -
          Field Original Value New Value
          Epic Link IGBF-1765 [ 17855 ]
          Hide
          nfreese Nowlan Freese added a comment - - edited

          The UCSC DAS data provider has been disabled as far back as release-9.0.1 in the igbDefaultPrefs.json (9.0.1 was as far back as I tested). In release-9.1.6 we removed DAS v1 support (IGBF-2474) and also removed DAS from igbDefaultPrefs.json. The removal of the DAS v1 module in IGB caused a few features to fail in IGB (external view tab), and this is what we received user requests for (HELP-374). We then restored DAS v1 support in IGB for release-9.1.8 (IGBF-2793), but we kept the DAS data provider as disabled. We then switched the DAS data provider to be on by default in release-9.1.10 (IGBF-3068).

          For testing, I went back to the release-9.1.0 branch (DAS disabled by default), enabled DAS, closed IGB, started release-9.1.2 branch (DAS disabled by default), and DAS remained enabled.

          My conclusion is that if a data provider is enabled, switching the data provider to disabled in igbDefaultPrefs.json will not override the enabled status of the data provider when a user switches to a new version of IGB. The user help issues related to DAS were due to a previous reliance on DAS for the external view tab, that has since been updated to use the UCSC REST API endpoint (IGBF-2863).

          My recommendation would be to remove the DAS data provider from igbDefaultPrefs.json. The UCSC REST API is providing all of the functionality that DAS provided, and we have only had das enabled by default for the last two releases (9.1.10 and 10.0.0). We can add documentation to the IGB user's guide for how to re-enable the DAS data provider, should a user need it.

          Show
          nfreese Nowlan Freese added a comment - - edited The UCSC DAS data provider has been disabled as far back as release-9.0.1 in the igbDefaultPrefs.json (9.0.1 was as far back as I tested). In release-9.1.6 we removed DAS v1 support ( IGBF-2474 ) and also removed DAS from igbDefaultPrefs.json. The removal of the DAS v1 module in IGB caused a few features to fail in IGB (external view tab), and this is what we received user requests for (HELP-374). We then restored DAS v1 support in IGB for release-9.1.8 ( IGBF-2793 ), but we kept the DAS data provider as disabled. We then switched the DAS data provider to be on by default in release-9.1.10 ( IGBF-3068 ). For testing, I went back to the release-9.1.0 branch (DAS disabled by default), enabled DAS, closed IGB, started release-9.1.2 branch (DAS disabled by default), and DAS remained enabled. My conclusion is that if a data provider is enabled, switching the data provider to disabled in igbDefaultPrefs.json will not override the enabled status of the data provider when a user switches to a new version of IGB. The user help issues related to DAS were due to a previous reliance on DAS for the external view tab, that has since been updated to use the UCSC REST API endpoint ( IGBF-2863 ). My recommendation would be to remove the DAS data provider from igbDefaultPrefs.json. The UCSC REST API is providing all of the functionality that DAS provided, and we have only had das enabled by default for the last two releases (9.1.10 and 10.0.0). We can add documentation to the IGB user's guide for how to re-enable the DAS data provider, should a user need it.
          Hide
          nfreese Nowlan Freese added a comment -
          • Changing "name" from UCSC to UCSC DAS with status "disabled" - still enabled in 10.1.0
          • Changing "name" from UCSC to UCSC DAS and defaultDataProviderId from 15 to 99 with status "disabled" - still enabled in 10.1.0
          • Changing "name" from UCSC to UCSC DAS and defaultDataProviderId from 15 to 99 and "loadPriority" to 99 with "status" set to disabled - still enabled in 10.1.0
          • Changing "url" but leaving everything else the same will cause the status to become disabled. So the "key" value is the url.

          The UCSC DAS data provider is stored in the user's Preferences, and is set to "Initialized" coming from version 10.0.1. In DataProviderManager.java it looks like the initializeDataProvider method is looking to see if the data provider is set to Disabled/Initialized. As UCSC DAS was already set to Initialized it is initializing it. So it does not appear that the current logic looks to see if there was a change in the status (enabled/disabled) before initializing, if the data provider was previously included in the user's preferences, and based on the above testing, the "key" is the url for the data provider.

          I'm wary of attempting to alter the logic the DataProviderManager, as the logic is pretty extensive and will delay the release. My preference remains removing DAS from the igbDefaultPrefs.json file, as it can be quickly added back by a user. An alternative would be to leave DAS as disabled and move ahead with this release, though this will impact all users as IGB startup will be slower. We could then remove DAS from the igbDefaultPrefs in a future release.

          Show
          nfreese Nowlan Freese added a comment - Changing "name" from UCSC to UCSC DAS with status "disabled" - still enabled in 10.1.0 Changing "name" from UCSC to UCSC DAS and defaultDataProviderId from 15 to 99 with status "disabled" - still enabled in 10.1.0 Changing "name" from UCSC to UCSC DAS and defaultDataProviderId from 15 to 99 and "loadPriority" to 99 with "status" set to disabled - still enabled in 10.1.0 Changing "url" but leaving everything else the same will cause the status to become disabled. So the "key" value is the url. The UCSC DAS data provider is stored in the user's Preferences, and is set to "Initialized" coming from version 10.0.1. In DataProviderManager.java it looks like the initializeDataProvider method is looking to see if the data provider is set to Disabled/Initialized. As UCSC DAS was already set to Initialized it is initializing it. So it does not appear that the current logic looks to see if there was a change in the status (enabled/disabled) before initializing, if the data provider was previously included in the user's preferences, and based on the above testing, the "key" is the url for the data provider. I'm wary of attempting to alter the logic the DataProviderManager, as the logic is pretty extensive and will delay the release. My preference remains removing DAS from the igbDefaultPrefs.json file, as it can be quickly added back by a user. An alternative would be to leave DAS as disabled and move ahead with this release, though this will impact all users as IGB startup will be slower. We could then remove DAS from the igbDefaultPrefs in a future release.
          nfreese Nowlan Freese made changes -
          Assignee Jaya Sravani Sirigineedi [ jsirigin ]
          nfreese Nowlan Freese made changes -
          Story Points 0.25 1
          ann.loraine Ann Loraine made changes -
          Sprint Fall 1 [ 202 ] Fall 1, Fall 2 [ 202, 203 ]
          ann.loraine Ann Loraine made changes -
          Rank Ranked higher
          jsirigin Jaya Sravani Sirigineedi made changes -
          Status To-Do [ 10305 ] In Progress [ 3 ]
          Hide
          jsirigin Jaya Sravani Sirigineedi added a comment -

          After a couple of hours of investigation, I found that there is a logic that is expected to remove a dataProvider whose defaultDataProviderId isn't present in the new igbDefaultPrefs.json but the logic is incorrect. As Nowlan Freese explained in the above comment the URL is the key, so even though we change the defaultDataProviderId to disable it, when we load the igbDefaultPrefs it just overwrites the old stored providerId in the user preferences, and again when we load the user preferences it overwrites the status of it. Also, we can't keep the defaultDataProviderId as a key as it is null for the custom user added providers. If we want to make changes to this logic and make it work, it might take some effort to do that. So, the easiet way possible for now is to either change the URL to disable it (which I think is not a proper way to do) or remove it completely from the igbDefaultPrefs file.

          Show
          jsirigin Jaya Sravani Sirigineedi added a comment - After a couple of hours of investigation, I found that there is a logic that is expected to remove a dataProvider whose defaultDataProviderId isn't present in the new igbDefaultPrefs.json but the logic is incorrect. As Nowlan Freese explained in the above comment the URL is the key, so even though we change the defaultDataProviderId to disable it, when we load the igbDefaultPrefs it just overwrites the old stored providerId in the user preferences, and again when we load the user preferences it overwrites the status of it. Also, we can't keep the defaultDataProviderId as a key as it is null for the custom user added providers. If we want to make changes to this logic and make it work, it might take some effort to do that. So, the easiet way possible for now is to either change the URL to disable it (which I think is not a proper way to do) or remove it completely from the igbDefaultPrefs file.
          Hide
          jsirigin Jaya Sravani Sirigineedi added a comment -

          As discussed, we will go ahead and remove the DAS provider from the igbDefaultPrefs to disable it for all users, if doesn't do this it will still be enabled for the old users and IGB startup takes time. We can find a way to make the logic work, but it needs some investigation, and as the code changes impact the way how IGB stores the preferences, it might need regressive testing, which won't be a good option to do before the release, so created a new ticket to investigate and implement this after the release: https://jira.bioviz.org/browse/IGBF-3905.

          Show
          jsirigin Jaya Sravani Sirigineedi added a comment - As discussed, we will go ahead and remove the DAS provider from the igbDefaultPrefs to disable it for all users, if doesn't do this it will still be enabled for the old users and IGB startup takes time. We can find a way to make the logic work, but it needs some investigation, and as the code changes impact the way how IGB stores the preferences, it might need regressive testing, which won't be a good option to do before the release, so created a new ticket to investigate and implement this after the release: https://jira.bioviz.org/browse/IGBF-3905 .
          jsirigin Jaya Sravani Sirigineedi made changes -
          Attachment UCSC_properties_info.png [ 18505 ]
          jsirigin Jaya Sravani Sirigineedi made changes -
          Attachment UCSC_Das_provider_removal.png [ 18506 ]
          Hide
          jsirigin Jaya Sravani Sirigineedi added a comment -

          Also, I have found the cause for the defaultDataProviderId not being shown in the properties info of the UCSC DataProvider it's happening because when I created a new constructor to add the datasetLinkoutUrl, I didn't add the id in the constructor. Fixed this issue as well. Code changes are done and available at https://bitbucket.org/jaya-sravani/integrated-genome-browser/branch/IGBF-3898. Tested the application and everything's working as expected. Please review and let me know if there are any issues.

          Below are the screenshots and the steps to test the fixes:

          • Run a previous version of IGB first, have UCSC Das enabled in it, and close the application.
          • Now, start IGB from the provided branch.
          • Click on configure in the Available data window.
          • You should test two things now
          • First, see whether the UCSC Das Data Provider is visible or not. It shouldn't be available anymore.
          • Second, click on the info icon in the UCSC Rest row and see whether the defaultDataProviderId is available or not. It should be available.
          Show
          jsirigin Jaya Sravani Sirigineedi added a comment - Also, I have found the cause for the defaultDataProviderId not being shown in the properties info of the UCSC DataProvider it's happening because when I created a new constructor to add the datasetLinkoutUrl, I didn't add the id in the constructor. Fixed this issue as well. Code changes are done and available at https://bitbucket.org/jaya-sravani/integrated-genome-browser/branch/IGBF-3898 . Tested the application and everything's working as expected. Please review and let me know if there are any issues. Below are the screenshots and the steps to test the fixes: Run a previous version of IGB first, have UCSC Das enabled in it, and close the application. Now, start IGB from the provided branch. Click on configure in the Available data window. You should test two things now First, see whether the UCSC Das Data Provider is visible or not. It shouldn't be available anymore. Second, click on the info icon in the UCSC Rest row and see whether the defaultDataProviderId is available or not. It should be available.
          jsirigin Jaya Sravani Sirigineedi made changes -
          Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
          jsirigin Jaya Sravani Sirigineedi made changes -
          Assignee Jaya Sravani Sirigineedi [ jsirigin ] Nowlan Freese [ nfreese ]
          jsirigin Jaya Sravani Sirigineedi made changes -
          Assignee Nowlan Freese [ nfreese ]
          nfreese Nowlan Freese made changes -
          Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
          Hide
          nfreese Nowlan Freese added a comment -

          Testing Sravani's branch on Mac.

          DAS does not appear in the Data Sources tab, and when I click on the info icon for UCSC REST there is a default data provider id.

          Code review looks good, though I'm unsure of why lines 243-246 were removed from DataProviderManager.java? Do we need this change to see the default data provider id?

          Show
          nfreese Nowlan Freese added a comment - Testing Sravani's branch on Mac. DAS does not appear in the Data Sources tab, and when I click on the info icon for UCSC REST there is a default data provider id. Code review looks good, though I'm unsure of why lines 243-246 were removed from DataProviderManager.java? Do we need this change to see the default data provider id?
          nfreese Nowlan Freese made changes -
          Assignee Jaya Sravani Sirigineedi [ jsirigin ]
          nfreese Nowlan Freese made changes -
          Status First Level Review in Progress [ 10301 ] To-Do [ 10305 ]
          jsirigin Jaya Sravani Sirigineedi made changes -
          Status To-Do [ 10305 ] In Progress [ 3 ]
          Hide
          jsirigin Jaya Sravani Sirigineedi added a comment - - edited

          After going through the code a bit more for this ticket, I understood that this part of the code is only executed when the user creates a custom data provider, which won't have the option to specify datasetLinkoutUrl and also won't have any defaultDataProviderId. So, the if condition will always be false and isn't required at all. That's why I have removed it and it's the code I added when adding the datasetLinkoutUrl to the constructors.

          Show
          jsirigin Jaya Sravani Sirigineedi added a comment - - edited After going through the code a bit more for this ticket, I understood that this part of the code is only executed when the user creates a custom data provider, which won't have the option to specify datasetLinkoutUrl and also won't have any defaultDataProviderId. So, the if condition will always be false and isn't required at all. That's why I have removed it and it's the code I added when adding the datasetLinkoutUrl to the constructors.
          jsirigin Jaya Sravani Sirigineedi made changes -
          Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
          jsirigin Jaya Sravani Sirigineedi made changes -
          Assignee Jaya Sravani Sirigineedi [ jsirigin ] Nowlan Freese [ nfreese ]
          Hide
          nfreese Nowlan Freese added a comment -

          Ah gotcha, thank you!

          Please go ahead and create a PR.

          Show
          nfreese Nowlan Freese added a comment - Ah gotcha, thank you! Please go ahead and create a PR.
          nfreese Nowlan Freese made changes -
          Assignee Nowlan Freese [ nfreese ] Jaya Sravani Sirigineedi [ jsirigin ]
          nfreese Nowlan Freese made changes -
          Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
          nfreese Nowlan Freese made changes -
          Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
          Show
          jsirigin Jaya Sravani Sirigineedi added a comment - Raised the pull request: https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/1036
          jsirigin Jaya Sravani Sirigineedi made changes -
          Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
          jsirigin Jaya Sravani Sirigineedi made changes -
          Assignee Jaya Sravani Sirigineedi [ jsirigin ] Nowlan Freese [ nfreese ]
          nfreese Nowlan Freese made changes -
          Assignee Nowlan Freese [ nfreese ] Ann Loraine [ aloraine ]
          ann.loraine Ann Loraine made changes -
          Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
          ann.loraine Ann Loraine made changes -
          Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
          ann.loraine Ann Loraine made changes -
          Assignee Ann Loraine [ aloraine ]
          Hide
          ann.loraine Ann Loraine added a comment -

          PR is merged and new installers built and copied to "early access" section of Bioviz.org website.
          Ready for testing.

          Show
          ann.loraine Ann Loraine added a comment - PR is merged and new installers built and copied to "early access" section of Bioviz.org website. Ready for testing.
          nfreese Nowlan Freese made changes -
          Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
          Hide
          nfreese Nowlan Freese added a comment -

          Testing using main branch installer on Mac.

          After running 10.0.1 and then switching to 10.1.0 the DAS data provider is no longer a Data Source.
          I am able to add back the DAS data provider in the Data Sources tab.
          The UCSC REST data provider id now appears in the "info" window in the Data Source tab.

          Closing ticket.

          Show
          nfreese Nowlan Freese added a comment - Testing using main branch installer on Mac. After running 10.0.1 and then switching to 10.1.0 the DAS data provider is no longer a Data Source. I am able to add back the DAS data provider in the Data Sources tab. The UCSC REST data provider id now appears in the "info" window in the Data Source tab. Closing ticket.
          nfreese Nowlan Freese made changes -
          Assignee Jaya Sravani Sirigineedi [ jsirigin ]
          nfreese Nowlan Freese made changes -
          Resolution Done [ 10000 ]
          Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]
          nfreese Nowlan Freese made changes -
          Fix Version/s 10.1.0 [ 11000 ]

            People

            • Assignee:
              jsirigin Jaya Sravani Sirigineedi
              Reporter:
              nfreese Nowlan Freese
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: