Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-2964

Investigate: “This feature is not reachable” error

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Story Points:
      2
    • Sprint:
      Fall 4 2021 Sep 27 - Oct 8, Fall 5 2021 Oct 11 - Oct 22, Fall 6 2021 Oct 25 - Nov 5

      Description

      See comment in IGBF-2948:

      IGB reports “This feature is not reachable” error even though the file is reachable in a Web browser.

        Attachments

          Issue Links

            Activity

            ann.loraine Ann Loraine created issue -
            ann.loraine Ann Loraine made changes -
            Field Original Value New Value
            Epic Link IGBF-2831 [ 19524 ]
            ann.loraine Ann Loraine made changes -
            Link This issue relates to IGBF-2948 [ IGBF-2948 ]
            ann.loraine Ann Loraine made changes -
            Sprint Fall 4 2021 Sep 27 - Oct 8 [ 130 ]
            ann.loraine Ann Loraine made changes -
            Sprint Fall 4 2021 Sep 27 - Oct 8 [ 130 ] Fall 4 2021 Sep 27 - Oct 8, Fall 5 2021 Oct 11 - Oct 22 [ 130, 131 ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            pbadzuh Philip Badzuh (Inactive) made changes -
            Assignee Philip Badzuh [ pbadzuh ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            pbadzuh Philip Badzuh (Inactive) added a comment - - edited
            • In this case of this issue, HttpURLConnection is returning an HTTP status code of -1, which indicates that 'no code can be discerned from the response (i.e., the response is not valid HTTP).'
            • The specific exception causing this is 'java.security.cert.CertificateException: No subject alternative DNS name matching {HOST}

              found.' This seems to be an SSL validation related issue, so we be able to avoid this error by altering the current HTTPS connection configuration.

            • More useful info here.
            Show
            pbadzuh Philip Badzuh (Inactive) added a comment - - edited In this case of this issue, HttpURLConnection is returning an HTTP status code of -1, which indicates that 'no code can be discerned from the response (i.e., the response is not valid HTTP).' The specific exception causing this is 'java.security.cert.CertificateException: No subject alternative DNS name matching {HOST} found.' This seems to be an SSL validation related issue, so we be able to avoid this error by altering the current HTTPS connection configuration. More useful info here .
            Hide
            pbadzuh Philip Badzuh (Inactive) added a comment -

            Please see my changes here. They prevent the error described in this issue by reducing SSL handshake verification stringency.

            Show
            pbadzuh Philip Badzuh (Inactive) added a comment - Please see my changes here . They prevent the error described in this issue by reducing SSL handshake verification stringency.
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Assignee Philip Badzuh [ pbadzuh ]
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Request:

            I'd like to better understand why certain data files are causing this problem.

            Can you please provide example(s) of where the server's certificate does not properly match the domain name mentioned in the data file URL, causing the SSL validation error?

            Could you indicate the particular method call where the exception is thrown? (Show a stacktrace?)

            This code seems risky because it appears to turn off all host verification:

            HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() {
                        public boolean verify(String hostname, SSLSession session) {
                            return true;
                        }
                   });   
            

            Is there a more targeted approach we can use instead?

            Show
            ann.loraine Ann Loraine added a comment - - edited Request: I'd like to better understand why certain data files are causing this problem. Can you please provide example(s) of where the server's certificate does not properly match the domain name mentioned in the data file URL, causing the SSL validation error? Could you indicate the particular method call where the exception is thrown? (Show a stacktrace?) This code seems risky because it appears to turn off all host verification: HttpsURLConnection.setDefaultHostnameVerifier( new HostnameVerifier() { public boolean verify( String hostname, SSLSession session) { return true ; } }); Is there a more targeted approach we can use instead?
            ann.loraine Ann Loraine made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ]
            ann.loraine Ann Loraine made changes -
            Status First Level Review in Progress [ 10301 ] To-Do [ 10305 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ] Philip Badzuh [ pbadzuh ]
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Returning to "To-Do" for Philip Badzuh to investigate further..

            Show
            ann.loraine Ann Loraine added a comment - - edited Returning to "To-Do" for Philip Badzuh to investigate further..
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            ann.loraine Ann Loraine made changes -
            Sprint Fall 4 2021 Sep 27 - Oct 8, Fall 5 2021 Oct 11 - Oct 22 [ 130, 131 ] Fall 4 2021 Sep 27 - Oct 8, Fall 5 2021 Oct 11 - Oct 22, Fall 6 2021 Oct 25 - Nov 5 [ 130, 131, 132 ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            pbadzuh Philip Badzuh (Inactive) added a comment -

            Sample problematic file:
            https://dashr2.lisanwanglab.org/tracks/hg38/DASHRv2.hg38.annot.pos.bigBed

            Inspecting the server's x509 SSL certificate when making a request for the above resource, I noticed that the subject common name was not dashr2.lisanwanglab.org as expected, but advp.lisanwanglab.org. The same was true for the subject alternative name field. It seems that for some reason the server is returning a cert for the wrong subdomain. What is even more confusing is that this behavior is inconsistent across clients. For example this behavior occurs when IGB makes the requests as well as when the openssl tool is used to view the certificate e.g.

            openssl s_client -showcerts -connect dashr2.lisanwanglab.org:443
            

            Making the same request using the browser or CURL , however, results in the proper certificate being returned by the server - with a common and alternative name of dashr2.lisanwanglab.org. It may be that under the the browser and CURL experience the same error, but fix it under the hood by making some sort of secondary request to the server for a different certificate matching the correct subdomain, however, I am unsure of how to check this.

            In any case, one potential solution to this issue could be to modify the HostnameVerifier I have implemented to allow all certificates that have the same domain and TLD, regardless of subdomain. I think this would both solve this issue and maintain security. Please let me know what you think [~aloraine].

            Show
            pbadzuh Philip Badzuh (Inactive) added a comment - Sample problematic file: https://dashr2.lisanwanglab.org/tracks/hg38/DASHRv2.hg38.annot.pos.bigBed Inspecting the server's x509 SSL certificate when making a request for the above resource, I noticed that the subject common name was not dashr2.lisanwanglab.org as expected, but advp.lisanwanglab.org. The same was true for the subject alternative name field. It seems that for some reason the server is returning a cert for the wrong subdomain. What is even more confusing is that this behavior is inconsistent across clients. For example this behavior occurs when IGB makes the requests as well as when the openssl tool is used to view the certificate e.g. openssl s_client -showcerts -connect dashr2.lisanwanglab.org:443 Making the same request using the browser or CURL , however, results in the proper certificate being returned by the server - with a common and alternative name of dashr2.lisanwanglab.org. It may be that under the the browser and CURL experience the same error, but fix it under the hood by making some sort of secondary request to the server for a different certificate matching the correct subdomain, however, I am unsure of how to check this. In any case, one potential solution to this issue could be to modify the HostnameVerifier I have implemented to allow all certificates that have the same domain and TLD, regardless of subdomain. I think this would both solve this issue and maintain security. Please let me know what you think [~aloraine] .
            Hide
            ann.loraine Ann Loraine added a comment -

            Thank you for the very clear discussion.
            I have a an additional questions regarding how the interaction between the server and client when the client is curl or a Web browser. How do you know that they are obtaining the correct certificate? Is it because the browser and curl accept the authenticity?

            Show
            ann.loraine Ann Loraine added a comment - Thank you for the very clear discussion. I have a an additional questions regarding how the interaction between the server and client when the client is curl or a Web browser. How do you know that they are obtaining the correct certificate? Is it because the browser and curl accept the authenticity?
            ann.loraine Ann Loraine made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            ann.loraine Ann Loraine made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            ann.loraine Ann Loraine made changes -
            Status First Level Review in Progress [ 10301 ] To-Do [ 10305 ]
            ann.loraine Ann Loraine made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            pbadzuh Philip Badzuh (Inactive) added a comment -

            Using the browser, you can inspect the certificate being used by clicking on the lock to the left of the URL. When using CURL, the -v flag can be passed for verbose output, which includes basic updates of the SSL handshake steps, including details about the certificate being used.

            Show
            pbadzuh Philip Badzuh (Inactive) added a comment - Using the browser, you can inspect the certificate being used by clicking on the lock to the left of the URL. When using CURL, the -v flag can be passed for verbose output, which includes basic updates of the SSL handshake steps, including details about the certificate being used.
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            I'm confused about there being multiple certificates returned by the site. My understanding from having requested and installed certificates is that you only get one! And you can often purchase a "star" certificate that will cover every subdomain, e.g., foo.bioviz.org or bar.bioviz.org would be validated the same "star" certificate.

            However, looking at the above certificates mentioned in the previous comments, it looks like the site is using a public service ("let's encrypt") that provides certificates for free, in some kind of dynamic way. I wonder if the site's configuration is messed up or incorrect in a minor way that affects IGB and openssl, but not the other applications, explaining why the site maintainers perhaps never noticed the problem. For this latter task, let's make a new ticket to investigate "let's encrypt".

            Maybe we ought to investigate how the "let's encrypt" configuration is done? Also, [~aloraine] is interested in how "let's encrypt" works in case we want to use for our Web applications, as well. Why? It's because the lab has multiple domains, for only a few of which do we currently purchase certificates, as they are costly. If we knew how to use "let's encrypt," we could provide SSL level security for every site, not just "bioviz" and the few others for which we purchase certificates.

            Show
            ann.loraine Ann Loraine added a comment - - edited I'm confused about there being multiple certificates returned by the site. My understanding from having requested and installed certificates is that you only get one! And you can often purchase a "star" certificate that will cover every subdomain, e.g., foo.bioviz.org or bar.bioviz.org would be validated the same "star" certificate. However, looking at the above certificates mentioned in the previous comments, it looks like the site is using a public service ("let's encrypt") that provides certificates for free, in some kind of dynamic way. I wonder if the site's configuration is messed up or incorrect in a minor way that affects IGB and openssl, but not the other applications, explaining why the site maintainers perhaps never noticed the problem. For this latter task, let's make a new ticket to investigate "let's encrypt". Maybe we ought to investigate how the "let's encrypt" configuration is done? Also, [~aloraine] is interested in how "let's encrypt" works in case we want to use for our Web applications, as well. Why? It's because the lab has multiple domains, for only a few of which do we currently purchase certificates, as they are costly. If we knew how to use "let's encrypt," we could provide SSL level security for every site, not just "bioviz" and the few others for which we purchase certificates.
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            To solve the immediate problem, I agree ([~aloraine]) that you should do this: "allow all certificates that have the same domain and TLD, regardless of subdomain".

            Make new tickets for these new tasks.

            Show
            ann.loraine Ann Loraine added a comment - - edited To solve the immediate problem, I agree ( [~aloraine] ) that you should do this: "allow all certificates that have the same domain and TLD, regardless of subdomain". Make new tickets for these new tasks.
            pbadzuh Philip Badzuh (Inactive) made changes -
            Link This issue relates to IGBF-3000 [ IGBF-3000 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Link This issue relates to IGBF-3001 [ IGBF-3001 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Assignee Philip Badzuh [ pbadzuh ]
            Hide
            pbadzuh Philip Badzuh (Inactive) added a comment -

            I have made new tickets for these tasks in IGBF-3000 and IGBF-3001. Closing this issue.

            Show
            pbadzuh Philip Badzuh (Inactive) added a comment - I have made new tickets for these tasks in IGBF-3000 and IGBF-3001 . Closing this issue.
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]
            pbadzuh Philip Badzuh (Inactive) made changes -
            Assignee Philip Badzuh [ pbadzuh ]

              People

              • Assignee:
                pbadzuh Philip Badzuh (Inactive)
                Reporter:
                ann.loraine Ann Loraine
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: