Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-3909

Add the ability to filter multiple values for Samtools tags

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 10.2.0
    • Labels:
      None

      Description

      In the Filter... feature, users can to add multiple filters for the tracks and these filters appear in a list in the Add\Remove\Edit filters window. When adding Samtools tags options, however, multiple filter options does not seem to work as expected.
      When adding a cell barcode(CR) filter, the track only shows reads associated with that CR. Then that filter was removed and another CR filter with a different barcode was added. The same thing happened: reads associated with the CR is displayed. That behavior is expected. However, when both the filters is present in the list, although the expected behavior is for the track to show reads from both the CR values, the observation is that all reads disappear.

        Attachments

          Issue Links

            Activity

            Hide
            karthik Karthik Raveendran added a comment -

            When multiple filters are added in the list, the logic only displays reads that meet all the filter conditions (AND operation). With Samtools tags, we need an OR operator which displays reads that meet atleast one of the conditions.

            Show
            karthik Karthik Raveendran added a comment - When multiple filters are added in the list, the logic only displays reads that meet all the filter conditions (AND operation). With Samtools tags, we need an OR operator which displays reads that meet atleast one of the conditions.
            Hide
            karthik Karthik Raveendran added a comment - - edited

            As the user may add several Samtool tags options in the list to filter out, UX changes in the Filter by window such as the ability to resize the window was added and the comparator option for the tags available now because it can be confusing to some users needed to be removed.

            Show
            karthik Karthik Raveendran added a comment - - edited As the user may add several Samtool tags options in the list to filter out, UX changes in the Filter by window such as the ability to resize the window was added and the comparator option for the tags available now because it can be confusing to some users needed to be removed.
            Hide
            karthik Karthik Raveendran added a comment -

            General Work Flow
            When the Filter by option is clicked on track dropdown menu,

            1. the actionPerformed() function in FilterAction object is triggered where the filter dialog (Add/Remove/Edit Filters dialog - see image) is created as well as an instance of the ConfigureFilters class is also created.
            2. When the Add button is clicked on, the addNew() function in that class is called where
            3. an instance of the ConfigureOptionsDialog (Add Filter dialog - see image) is created at the top of the function. The init() function in this class is where
            4. an instance of the ConfigureOptionPanel is created at the top of the function which is a panel where users can select a filter from the dropdown and UI elements associated with the filter is populated here.
            5. The init() function in this class is where the dropdown menu for the filters are created and listeners are added to each menu item. By default, the filter on the top the dropdown list is selected (at the time of writing this comment it is Average Quality Score) and this triggers the listeners to populate options, such as, average_quality textfield as well as a comparator dropdown menu. Let's call these Filter Options
            6. When the filter dropdown menu item in Add Filter dialog is selected, the change listener will call the setSelected() is called where after checking there are no null values initParamPanel() is called which in turn calls addOptions()
            7. Here, the Filter Options are added to the ConfigureOptionPanel and the respective Filter Options listeners, such as, a change listener when a value is added to a textfield which sets the parameter in the paramMap which key-value Map for all the Filter Options. All of this happens in the ConfigureOptionPanel .
            8. When the user submits their choice (click the OK button on Add Filter dialog), the values of the Filter Options is added to the ConfigureOptionsDialog list. The values are compared with the track properties in FilterSymmerty() in the SamtoolsFilter
            Show
            karthik Karthik Raveendran added a comment - General Work Flow When the Filter by option is clicked on track dropdown menu, the actionPerformed() function in FilterAction object is triggered where the filter dialog ( Add/Remove/Edit Filters dialog - see image) is created as well as an instance of the ConfigureFilters class is also created. When the Add button is clicked on, the addNew() function in that class is called where an instance of the ConfigureOptionsDialog ( Add Filter dialog - see image) is created at the top of the function. The init() function in this class is where an instance of the ConfigureOptionPanel is created at the top of the function which is a panel where users can select a filter from the dropdown and UI elements associated with the filter is populated here. The init() function in this class is where the dropdown menu for the filters are created and listeners are added to each menu item. By default, the filter on the top the dropdown list is selected (at the time of writing this comment it is Average Quality Score ) and this triggers the listeners to populate options, such as, average_quality textfield as well as a comparator dropdown menu. Let's call these Filter Options When the filter dropdown menu item in Add Filter dialog is selected, the change listener will call the setSelected() is called where after checking there are no null values initParamPanel() is called which in turn calls addOptions() Here, the Filter Options are added to the ConfigureOptionPanel and the respective Filter Options listeners, such as, a change listener when a value is added to a textfield which sets the parameter in the paramMap which key-value Map for all the Filter Options . All of this happens in the ConfigureOptionPanel . When the user submits their choice (click the OK button on Add Filter dialog), the values of the Filter Options is added to the ConfigureOptionsDialog list. The values are compared with the track properties in FilterSymmerty() in the SamtoolsFilter
            Hide
            karthik Karthik Raveendran added a comment -

            Currently, multiple filter values can be added to the samtags filters using a semi-colon separator. See Commit

            Show
            karthik Karthik Raveendran added a comment - Currently, multiple filter values can be added to the samtags filters using a semi-colon separator. See Commit
            Hide
            nfreese Nowlan Freese added a comment -

            Tested using Karthik's branch on Mac.

            Filter by multiple values is working well. I have a couple of thoughts/changes:

            • I think it would be good to separate your commit into two commits, 1) adding the ability to filter multiple samtools tag values, i.e. the changes to SamTagsFilter.java, and 2) improving the filters window, i.e. the changes to FilterAction.java and ConfigureFilters.java.
            • Can we avoid the import of java.awt.* (i.e. don't import .*)
            • Add back the comparator for not equal. That way I can put in two or three CR values and then choose if I want to only see these CR values, or I want to not see these CR values.
            Show
            nfreese Nowlan Freese added a comment - Tested using Karthik's branch on Mac. Filter by multiple values is working well. I have a couple of thoughts/changes: I think it would be good to separate your commit into two commits, 1) adding the ability to filter multiple samtools tag values, i.e. the changes to SamTagsFilter.java, and 2) improving the filters window, i.e. the changes to FilterAction.java and ConfigureFilters.java. Can we avoid the import of java.awt.* (i.e. don't import .*) Add back the comparator for not equal. That way I can put in two or three CR values and then choose if I want to only see these CR values, or I want to not see these CR values.
            Hide
            karthik Karthik Raveendran added a comment -

            I was able to remove java.awt.* and add the not equals comparator to the logic. See commit

            Show
            karthik Karthik Raveendran added a comment - I was able to remove java.awt.* and add the not equals comparator to the logic. See commit
            Hide
            karthik Karthik Raveendran added a comment -

            Testing Protocol:

            Open IGB and...

            ...add 5k Human PBMCs, 3' v3.1, Chromium Controller.zip quickload as a data source

            a) Unzip the file on your local machine
            b) On IGB, Press Alt+P and choose the Data Sources Tab
            c) Click on Add... : Use "5k Human PBMCs" as the name of the server and choose the unzipped folder
            d) Hit submit and then OK
            e) Choose Homo Sapiens from the carousel and select the only file in 5k Human PBMCs from the Available data section. Do NOT load the data

            1. Search for GNLY at search bar at the top-left of IGB and load data
            2. Right-click on any of the reads and select Get info and copy the CR value
            3. Repeat Step 2 two or more times and paste the values in a text file, separated by semi-colons
            4. Or use the following values for step 8: CACTGAAAGCTGTCCG;TCTCTGGTCAAGAGTA;ACTGCAAGTCAGATTC
            5. Right-click on the track name and choose Filter.. from the menu options
            6. In the Add/Remove/Edit Filters window, click on Add button
            7. In the Add Filter window, choose Samtools Tags from the Show only dropdown
            8. Select CR for tag, add the values, choose "=" for comparator and hit OK
            9. The filter you create should appear in the list on Add/Remove/Edit Filters. If so, hit Ok
            10. Ideally, all reads except the ones with CR values you chose will disappear
            11. Repeat Step 5 and remove the filter you created and continue with the rest of the steps
            12. On step 8, instead of "=" choose the not equals option in the comparator and continue on
            13. The changes on the track you see after this steps maybe subtle, please look for any change in the track.
            Show
            karthik Karthik Raveendran added a comment - Testing Protocol: Open IGB and... ...add 5k Human PBMCs, 3' v3.1, Chromium Controller.zip quickload as a data source a) Unzip the file on your local machine b) On IGB, Press Alt+P and choose the Data Sources Tab c) Click on Add... : Use "5k Human PBMCs" as the name of the server and choose the unzipped folder d) Hit submit and then OK e) Choose Homo Sapiens from the carousel and select the only file in 5k Human PBMCs from the Available data section. Do NOT load the data Search for GNLY at search bar at the top-left of IGB and load data Right-click on any of the reads and select Get info and copy the CR value Repeat Step 2 two or more times and paste the values in a text file, separated by semi-colons Or use the following values for step 8: CACTGAAAGCTGTCCG;TCTCTGGTCAAGAGTA;ACTGCAAGTCAGATTC Right-click on the track name and choose Filter.. from the menu options In the Add/Remove/Edit Filters window, click on Add button In the Add Filter window, choose Samtools Tags from the Show only dropdown Select CR for tag, add the values, choose "=" for comparator and hit OK The filter you create should appear in the list on Add/Remove/Edit Filters. If so, hit Ok Ideally, all reads except the ones with CR values you chose will disappear Repeat Step 5 and remove the filter you created and continue with the rest of the steps On step 8, instead of "=" choose the not equals option in the comparator and continue on The changes on the track you see after this steps maybe subtle, please look for any change in the track.
            Hide
            pkulzer Paige Kulzer (Inactive) added a comment -

            I'm running into an issue during step 7: there's no Samtools Tags option present in the Show Only dropdown menu. In the list I'm seeing, it goes from "Run number" to "Score". For reference, I'm running the newly released version of IGB 10.1.0.

            Show
            pkulzer Paige Kulzer (Inactive) added a comment - I'm running into an issue during step 7: there's no Samtools Tags option present in the Show Only dropdown menu. In the list I'm seeing, it goes from "Run number" to "Score". For reference, I'm running the newly released version of IGB 10.1.0.
            Hide
            karthik Karthik Raveendran added a comment - - edited

            I am sorry, you will have to pull the branch from my repo and run that to test it. See commit

            Show
            karthik Karthik Raveendran added a comment - - edited I am sorry, you will have to pull the branch from my repo and run that to test it. See commit
            Hide
            pkulzer Paige Kulzer (Inactive) added a comment -

            Thanks, Karthik – that slipped my mind!

            Following the above protocol, I confirmed that the Samtools Tag filter is functioning as expected. There were no errors in the Log. For reference, here is the list I used: TACAGGTCAGATCCTA;TTTAGTCTCCGGACTG;ACATCCCCACAGTGTT

            Show
            pkulzer Paige Kulzer (Inactive) added a comment - Thanks, Karthik – that slipped my mind! Following the above protocol, I confirmed that the Samtools Tag filter is functioning as expected. There were no errors in the Log. For reference, here is the list I used: TACAGGTCAGATCCTA;TTTAGTCTCCGGACTG;ACATCCCCACAGTGTT
            Show
            karthik Karthik Raveendran added a comment - Pull Request Submitted: https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/1053
            Hide
            ann.loraine Ann Loraine added a comment -

            PR is merged into main branch, new main branch installers are built, and new installers are deployed to early access section of bioviz.org.

            Ready for testing.

            Show
            ann.loraine Ann Loraine added a comment - PR is merged into main branch, new main branch installers are built, and new installers are deployed to early access section of bioviz.org. Ready for testing.
            Hide
            uchinta Udaya Chinta (Inactive) added a comment -

            I followed the testing protocol and tested the changes. The Samtools tag filter functionality is working as expected. The reference CR I used for testing is TGAGGAGCATGTTCAG;GATAGAATCTTGGGCG;TGTAACGTCATCTCTA.

            I observed that users can still add another Samtools tag. If we provide any CR value to the second Samtools tag, the behavior is inconsistent—sometimes we get reads, and sometimes we do not. I tested with different combinations of comparators in the first and second Samtools tags and the issue persists.. For two Samtools tags, changes are needed.

            I confirm that a single Samtools tag is working as expected.

            Show
            uchinta Udaya Chinta (Inactive) added a comment - I followed the testing protocol and tested the changes. The Samtools tag filter functionality is working as expected. The reference CR I used for testing is TGAGGAGCATGTTCAG;GATAGAATCTTGGGCG;TGTAACGTCATCTCTA. I observed that users can still add another Samtools tag. If we provide any CR value to the second Samtools tag, the behavior is inconsistent—sometimes we get reads, and sometimes we do not. I tested with different combinations of comparators in the first and second Samtools tags and the issue persists.. For two Samtools tags, changes are needed. I confirm that a single Samtools tag is working as expected.
            Hide
            uchinta Udaya Chinta (Inactive) added a comment - - edited

            I tested below scenarios:

            1. Adding and Editing a Single Samtools Tag:
              a. Add a CR value as TGAGGAGCATGTTCAG;GATAGAATCTTGGGCG;TGTAACGTCATCTCTA and select the "equal" comparator. Check the filter.
              b. Edit the already added Samtools tag by selecting the "edit" button. Change the CR value to TGAGGAGCATGTTCAG;GATAGAATCTTGGGCG, keeping the comparator as "equal." Click "OK" and check the filter. It does not display any reads.
              Observation: To make the edited filter work, the existing Samtools tag filter must be removed, and a new one needs to be added.
            2. Adding Multiple Samtools Tag Filters
              a. Remove the previously added filter using the "remove" button. Add a new Samtools tag filter, set the CR value as TGAGGAGCATGTTCAG;GATAGAATCTTGGGCG, and choose the "equal" comparator. Click "OK" and verify the filter. Reads are displayed.
              b. With the above filter active, add another Samtools tag filter. Set the CR value as TGTAACGTCATCTCTA and choose the "equal" comparator. Click "OK" and check both filters. No reads are displayed.
              Is this behavior expected?
            3. Editing the Second Filter
              a. Edit the second filter, change the comparator to "not equal," and click "OK." Verify the reads. The results show CR values matching TGAGGAGCATGTTCAG; GATAGAATCTTGGGCG while excluding TGTAACGTCATCTCTA and all other CR values available, as expected.
            4. Editing the First Filter to "Not Equal"
              a. Edit the first filter and set the comparator to "not equal." Reads with CR values other than TGAGGAGCATGTTCAG;GATAGAATCTTGGGCG;TGTAACGTCATCTCTA are displayed, as expected.
            5. Editing the First Filter Back to "Equal"
              a. Edit the first filter and set the comparator back to "equal." No reads are displayed. This case and step 3 are same but this behavior differs from Step 3, where a response was received.
              An exception was logged. The log file is attached.
              SamtoolTag_Step6.txt
            6. Changing the Second Filter to "Not Equal"
              a. Now change the comparator for the second filter to "not equal" and verify the filter. No response is received, which is expected. This is similar to step 2. However, the logged exception persists. The log file is attached.
              SamtoolTag_Step7.txt
            7. Editing Both Filters to "Not Equal"
              a. Edit both filters and set their comparators to "not equal." The response is received as expected.
            Show
            uchinta Udaya Chinta (Inactive) added a comment - - edited I tested below scenarios: Adding and Editing a Single Samtools Tag: a. Add a CR value as TGAGGAGCATGTTCAG;GATAGAATCTTGGGCG;TGTAACGTCATCTCTA and select the "equal" comparator. Check the filter. b. Edit the already added Samtools tag by selecting the "edit" button. Change the CR value to TGAGGAGCATGTTCAG;GATAGAATCTTGGGCG, keeping the comparator as "equal." Click "OK" and check the filter. It does not display any reads. Observation: To make the edited filter work, the existing Samtools tag filter must be removed, and a new one needs to be added. Adding Multiple Samtools Tag Filters a. Remove the previously added filter using the "remove" button. Add a new Samtools tag filter, set the CR value as TGAGGAGCATGTTCAG;GATAGAATCTTGGGCG, and choose the "equal" comparator. Click "OK" and verify the filter. Reads are displayed. b. With the above filter active, add another Samtools tag filter. Set the CR value as TGTAACGTCATCTCTA and choose the "equal" comparator. Click "OK" and check both filters. No reads are displayed. Is this behavior expected? Editing the Second Filter a. Edit the second filter, change the comparator to "not equal," and click "OK." Verify the reads. The results show CR values matching TGAGGAGCATGTTCAG; GATAGAATCTTGGGCG while excluding TGTAACGTCATCTCTA and all other CR values available, as expected. Editing the First Filter to "Not Equal" a. Edit the first filter and set the comparator to "not equal." Reads with CR values other than TGAGGAGCATGTTCAG;GATAGAATCTTGGGCG;TGTAACGTCATCTCTA are displayed, as expected. Editing the First Filter Back to "Equal" a. Edit the first filter and set the comparator back to "equal." No reads are displayed. This case and step 3 are same but this behavior differs from Step 3, where a response was received. An exception was logged . The log file is attached. SamtoolTag_Step6.txt Changing the Second Filter to "Not Equal" a. Now change the comparator for the second filter to "not equal" and verify the filter. No response is received, which is expected. This is similar to step 2. However, the logged exception persists. The log file is attached. SamtoolTag_Step7.txt Editing Both Filters to "Not Equal" a. Edit both filters and set their comparators to "not equal." The response is received as expected.
            Hide
            uchinta Udaya Chinta (Inactive) added a comment -

            Dr.Nowlan Freese, As discussed, I have added all the tested scenarios to this ticket. Please review them.

            Show
            uchinta Udaya Chinta (Inactive) added a comment - Dr. Nowlan Freese , As discussed, I have added all the tested scenarios to this ticket. Please review them.
            Hide
            nfreese Nowlan Freese added a comment - - edited

            I followed Udaya's testing instructions:
            Test 1. I saw the same result, no reads are displayed. This is incorrect as editing the CR values for the filter should still show reads. Seems like the edit button may not be working correctly.
            Test 2. This is the expected behavior.
            Test 5. I'm not seeing an exception, but no reads are showing, which is not the expected behavior. If one filter is set to not equal a CR value and another filter is set to equal a CR value, it should show reads.

            Karthik Raveendran - Can you take a quick look at this? If this seems like a new issue lets push this ticket back to to-do.

            Good catch by Udaya Chinta, I had not caught this in my testing.

            Show
            nfreese Nowlan Freese added a comment - - edited I followed Udaya's testing instructions: Test 1. I saw the same result, no reads are displayed. This is incorrect as editing the CR values for the filter should still show reads. Seems like the edit button may not be working correctly. Test 2. This is the expected behavior. Test 5. I'm not seeing an exception, but no reads are showing, which is not the expected behavior. If one filter is set to not equal a CR value and another filter is set to equal a CR value, it should show reads. Karthik Raveendran - Can you take a quick look at this? If this seems like a new issue lets push this ticket back to to-do. Good catch by Udaya Chinta , I had not caught this in my testing.
            Hide
            karthik Karthik Raveendran added a comment -

            The issue where not reads are shown after editing the filter values is because of the way case sensitivity is being handled. The user input was forced to be lower case after the filter was applied and saved in lower case in the session. When the user edits the value, the saved lower case values are compared to the read tag values which are in upper case.

            After discussing with Dr. Nowlan Freese, case sensitivity of the tag values should be handled in the final design potentially by giving the user a checkbox to click on. For this ticket a temporary fix has been implemented where values are not stored in lower case at all. See commit

            Show
            karthik Karthik Raveendran added a comment - The issue where not reads are shown after editing the filter values is because of the way case sensitivity is being handled. The user input was forced to be lower case after the filter was applied and saved in lower case in the session. When the user edits the value, the saved lower case values are compared to the read tag values which are in upper case. After discussing with Dr. Nowlan Freese , case sensitivity of the tag values should be handled in the final design potentially by giving the user a checkbox to click on. For this ticket a temporary fix has been implemented where values are not stored in lower case at all. See commit
            Hide
            uchinta Udaya Chinta (Inactive) added a comment -

            Tested all the previously mentioned test scenarios. All the scenario's are working good. Hence moving it to ready for pull request.

            Show
            uchinta Udaya Chinta (Inactive) added a comment - Tested all the previously mentioned test scenarios. All the scenario's are working good. Hence moving it to ready for pull request.
            Show
            karthik Karthik Raveendran added a comment - PR Submitted https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/1057
            Hide
            ann.loraine Ann Loraine added a comment -

            PR is merged.

            Regarding testing:

            • This changes aspects of filtering logic. For testing, should we check that other filters still work? For example, filtering based on whether annotations (e.g., sequence read alignments) contain gaps, or not.
            Show
            ann.loraine Ann Loraine added a comment - PR is merged. Regarding testing: This changes aspects of filtering logic. For testing, should we check that other filters still work? For example, filtering based on whether annotations (e.g., sequence read alignments) contain gaps, or not.
            Hide
            ann.loraine Ann Loraine added a comment -

            PR is merged and new installers are built and deployed to bioviz.org web site early access section.

            Ready for testing.

            Show
            ann.loraine Ann Loraine added a comment - PR is merged and new installers are built and deployed to bioviz.org web site early access section. Ready for testing.
            Hide
            pkulzer Paige Kulzer (Inactive) added a comment -

            Tested with main branch installer on Mac using Karthik's instructions above. I also walked through the various scenarios Udaya outlined and, per Dr. Loraine's suggestion, tested the functionality of several other filter options (e.g., Reads with gaps, Reads with no gaps, Score). Everything seems to be working correctly, no errors in the logs.

            Closing ticket.

            Show
            pkulzer Paige Kulzer (Inactive) added a comment - Tested with main branch installer on Mac using Karthik's instructions above. I also walked through the various scenarios Udaya outlined and, per Dr. Loraine's suggestion, tested the functionality of several other filter options (e.g., Reads with gaps, Reads with no gaps, Score). Everything seems to be working correctly, no errors in the logs. Closing ticket.

              People

              • Assignee:
                karthik Karthik Raveendran
                Reporter:
                karthik Karthik Raveendran
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: