The file does not pass this gff3 validator: http://genometools.org/cgi-bin/gff3validator.cgi

This file seems odd to me. Would need to do a comparison with the gff3 specifications to double-check the file formatting and compare that to the gff3 expectations of IGB.

Me and Prutha Kulkarni were working on this today. The following are our findings.
1) The gff3 validator mentioned in the above comment was unable to pass the HomoSapien gff3 file. The error message received was "Validation unsuccessful!
GenomeTools error: the child feature with type 'pseudogenic_transcript' on line 8 in file "/var/www/servers/genometools.org/htdocs/cgi-bin/gff3/GFF3_HS_only.gff3" is not part-of parent feature with type 'gene' given on line 3 (according to type checker 'OBO file /home/satta/genometools_for_web/gtdata/obo_files/so.obo')"
The file uploaded can be found in the link "http://igbquickload.org/smokeTestingQuickload/H_sapiens_Dec_2013/GFF3/"
So, we were not sure if we could rely on the validator for proper results.

2) Dr. Nowlan Freese provided me a reference to the format for .gff3 file.
https://github.com/The-Sequence-Ontology/Specifications/blob/master/gff3.md
It says that the ID field should be unique for all children.
"ID
Indicates the ID of the feature. The ID attribute is required for features that have children (e.g. gene and mRNAs), or for those that span multiple lines, but are optional for other features. IDs for each feature must be unique within the scope of the GFF file. In the case of discontinuous features (i.e. a single feature that exists over multiple genomic locations) the same ID may appear on multiple lines. All lines that share an ID must collectively represent a single feature."
The .gff file provided does not contain unique IDs. Also, it does not contain Parent attribute. We are not sure if this is alright, since the other .gff3 files we referred have Parent attribute.

3) We tried to run the test case for GFFParser using fimo.gff file provided, and got an error message as below:
"java.io.IOException: Using GFF symloader but GFF3 file detected."
So, we tried to run the test case for GFF3Parser. Initially, this test case was skipped using @Ignore annotation. We removed it and tried to check if the file is getting parsed properly or not.
We were getting Null Pointer Exception in 'testParseCanonical' function on line "testResults(gff3.getGenome())" in GFF3ParserTest. While debugging, we found out that the buildindex() function call in init() function inside getGenome() was causing NPE.
So we went ahead and tried to debug buildindex() in SymLoader.java and figured that the line "sortedResult = externalSortService.merge(input, uri.toString(), comparatorMetadata, conf);" is causing error. We are not getting any value in sortedResult and merge is a function inside externalSortService interface.
We are not sure how to find out what the function does as it is only called with parameters and does not have a function body.
Kindly suggest further steps so that we can continue working with this tomorrow.

cc: Prof. [~aloraine]

Here's the FIMO page where they include an example fimo.gff output: http://meme-suite.org/doc/fimo.html

I'm wondering if that gff3 validator I included in a previous comment is correct. May be worth checking another validator: http://www.raetschlab.org/suppl/gff-tools

Prof. [~aloraine], could you please contact the FIMO team and check with them if the format they've given is as per the specifications or not?

Also, we will check the other validator which is mentioned by Dr Nowlan Freese today.

Findings:

• In the test case for simplest version of GFF parser the code is written to parse the format of the file which is present in the "Resources/Data" directory and is not generic.
• For eg. assertEquals(99, sym.getStart());
• Here, the start position is hardcoded which can be different for every file. Likewise, there are many assert statements in the test case.
• Initially, we tried to change the code and make it work for this particular file but we realized that the entire code was very specific to the file which is present in "Resources/Data" directory.
• So instead, we tried to upload the file directly in IGB and attaching screenshots of the "selection info" and the changed file to the ticket.
  Shamika Gajanan Kulkarni and I are attaching screenshots for both the platforms i.e. MACOS and WINDOWS just to be sure.
• One thing we wanted to mention is the "selection info" is showing "=" for each tag value which is kind of weird. We don't know if any particular value is missing for the tag and that is why it is showing "=" where it was expecting something else. Because ideally, the value corresponding to the key should be extracted while parsing and not the value starting from "=".

Prof. [~aloraine], could you please let us know how to proceed further?

Can you look into the code to find out why it is showing the = character in the Selection Info values column?

Somehow the code is able to figure out what the property names are, e.g, "Name", "Qvalue" etc. So it must be using the "=" character somehow.

The code was using regex for parsing the file. So, we changed it to remove the "=" and now it is getting parsed properly.
Attaching a screenshot for future reference.
cc: [~aloraine] , Shamika Gajanan Kulkarni

This is a good writeup someone posted on Biostars detailing all of the different versions of GFF and GTF: https://github.com/NBISweden/GAAS/blob/master/annotation/knowledge/gxf.md

Please find the regex code changes in the link below:

https://bitbucket.org/skulka2710/shamika_igb/branch/IGBF-2381#diff

Philip Badzuh - Can you please submit a PR for Shamika? You can do it by forking the IGB repo, cloning it to your local, adding her fork as a remote, fetching her branch, pushing her branch to your fork, and then submitting a PR from there to the team repo master branch.

Her authorship of the commit will still be visible, by the way.

Please see the PR below:
https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/794/igbf-2381-fix-regex-to-not-display-in/diff

[~aloraine]

My recommendation would be to revert these changes.

The changes in the PR above do not appear to fix the issue. Testing on 9.1.6 on the attached fimo.gff file loads the data exactly the same as 9.1.4. The main issue is that only a small number of the data appear in IGB compared to examination of the file itself. However, the data that do appear have correct formatting.

To force IGB to show all of the data, the ##gff-version 3 first line comment can be removed from the fimo.gff file. This appears to cause IGB to parse the data not as a GFF3 file, but as a GTF file.

Note that GTF files parse the attribute column as: attribute_name “attribute_value”; attribute_name “attribute_value”;

whereas GFF3 files parse the attribute column as: ID=cds00004;Parent=mRNA00001,mRNA00002;Name=edenprotein.4

The attribute values in the fimo.gff file are formatted according to the GFF3 spec as: Name=NNNNKCMTYGRDWHYYGTG_1-;Alias=MEME-2;

So by removing ##gff-version 3 and loading the fimo.gff file as a GTF through IGB the attribute column values appear with the equals signs. By stripping the equals signs in the parsing this causes the fimo.gff file to appear to be loaded correctly (through the GTF parser), but it is affecting the parsing of GTF files.

In 9.1.6 the GTF_HomoSapienChr1Only.gtf file loads all of the data in a single row and the attribute column does not contain the correct values. In 9.1.4, the data appear correct visually, though the attribute column also does not appear to be parsed correctly.

After reading through this description of GTF/GFF file formats I found this sentence describing GFF3 ID: "The ID attribute is required for features that have children (e.g. gene and mRNAs), or for those that span multiple lines, but are optional for other features." The ID field is optional and the NAME field does not have to be unique: "Unlike IDs, there is no requirement that the Name be unique within the file."

I think it may be worth trying to understand further why the fimo.gff file is not displaying correctly in IGB. Edit: The fimo.gff file includes a non-unique ID attribute, which causes incorrect parsing of the file. Removal of the ID attribute from the fimo file appears to fix the issue.

I agree. We should revert. Should I go ahead and do that now?

Yes, I don't see any reason to leave the changes in master for the time being. I think this issue will need a much deeper dive into the current GFF/GTF parsing in IGB.

Reverted. Moving to Closed.

Dr. Loraine pointed out that the fimo.gff file does have an ID attribute in the 9th column. The ID must be unique: " IDs for each feature must be unique within the scope of the GFF file." Removing the ID attribute from each row appears to fix the file. All of the data are visible in IGB and the data appear to be parsed correctly (attached).

This seems to be an issue with the output of FIMO, and not an issue with how IGB is parsing the GFF3.