Jennifer - finished creating header for .bedgraph files. Moving to 'needs testing'.

I do see the header when exporting a bedgraph file, but I am not seeing all chromosomes saved when exporting a bedgraph. It seems that the bedgraph file that is exported only contains data for the chromosome the user is on when they export the bedgraph. So for example, if a user is on Chromosome 2, the exported bedgraph will only contain data for chromosome 2 but it should contain data for all chromosomes. If I am misunderstanding this story in some way, please let me know. For now, I am re-assigning the story to Jennifer and moving it to the To-Do column.

Note: This had not been merged into main repo master - it was moved to Needs Testing incorrectly. However, Ivory (see below) has noted some problems. She is now taking over this issue.

Assigning to Ivory as Jenny won't be able to get back to this before Friday.

I pulled branch IGBF-1090. I'm assessing the current state of the functionality.

In my first effort to test the current state, IGB froze when I tried to save a graph track.
The second time, it saved a track (named Chr1PartiallyLoaded.bedgraph.*.bedgraph) but it only saved a portion of the data that I had loaded and in view, not sure what determined the stop point.
I also loaded data on Chr2, and saved the track. Data is saved from BOTH Chr1 and Chr2. All loaded data from Chr2 was saved, but the Chr1 data still stops at 8,227,586, even thought there is data loaded all the way to 8,251,939.
Loaded nearly all of Chr1, saved. File has Chr1 data up to 8227586, and has Chr2 data.

Sometimes (not always) the default file time in the save track dialogue is .gr instead of .bedgraph.

Loaded whole genome, saved. Chr1 data still stops at 8227586. Chr2 data stops at 8233721 (the same place that the data was loaded to when I saved Chr2 for the first time).

Restarted IGB.
Loaded data here: Chr4:2,122,072-2,127,580. Zoomed out to here, loaded data: Chr4:2,110,482-2,139,204. Zoomed out to here (no load): Chr4:1,984,514-2,265,538. Saved. Saved data includes: Chr4:2,077,256-2,128,082. Starts just ahead of the left-est loaded data (good) but stops about 500 bp to the right of the first loaded right edge. That location would have been appropriate based on the first region loaded. But it seems the right boundary was not updated when more data was loaded.
Loaded: Chr4:1,677,694-2,863,442. Loaded Chr5:22,391,063-22,401,357. Saved.
Saved regions include: Chr4:1,672,970-2,128,082 and Chr5:22,390,711-22,401,935.
Loaded: Chr5:22,338,488-22,453,028,
Loaded: (center most) Chr2:14,515,589-14,522,725; Chr2:14,493,243-14,538,794; (outermost) Chr2:14,472,606-14,559,482
Loaded: (center) Chr1:19,251,864-19,268,401; (left) Chr1:17,845,180-17,881,741; (right) Chr1:25,918,164-26,094,571
Loaded: (all) ChrM:0-366,924
Saved region includes:
Chr1: Has all data from all three loaded regions.
Chr2: Has all loaded data.
Chr4: Has all loaded data.
Chr5: Has all loaded data.
ChrM: Has all loaded data.
So... it saved perfectly that time. Curious.

New test. I removed the graph file and started again (same file: BA1_2.sm.bedgraph.gz).

Loaded Chr1:24,484,001-24,488,309. saved. Whole region saved.
Loaded Chr1:24,475,962-24,496,348. saved. Whole region saved.
Loaded Chr2:12,528,756-12,542,799, then Chr2:12,410,980-12,638,925. saved. Whole region on both chromosome saved.

New test (new file: BA2.sm.bedgraph.gz)

Loaded Chr1:24,434,194-24,553,049. saved. Saved it all.
Loaded Chr1:24,039,165-24,878,915. saved. saved it all.
Loaded Chr1:24,039,165-24,878,915; Chr2:8,097,811-8,340,320. saved. saved it all (both chromosomes).

New Test (restarted IGB, now using BA3.sm.bedgraph.gz)

Loaded Chr1:7,650,931-7,659,240; Chr1:7,648,212-7,667,333. saved. loaded file.
IGB is in a weird state where it will not respond to mouse clicks. (you get the keys and the text box)
Restart IGB. looks like it didn't actually saved that file.
Saving a new file.... spinning pinwheel of death.

New Test (restarted IGB, now using C1.sm.bedgraph.gz)

Loaded Chr1:4,664,013-4,683,134. loaded Chr1:4,594,501-4,763,591. saved.
Loaded some on chr3, then loaded more, saved. both chromosomes saved. All is well.

New Test. Reset preferences.
loaded part of chr1, then more, saved, all saved. loaded more, saved, all saved.
Changed the load mode to Genome (did not change view). Saved.
Chr1 is saved up to the right boundry of the last save.
Chr2 is all saved. Chr3, nothing saved. Chr4 all saved. Chr5 nothing saved. ChrC and ChrM, all saved.
Saved again. all data from every chr is saved.

Ok, something somewhere doesn't work BUT it only doesn't work some of the time.
It may have to do with refreshing IGB's notion of what is currently loaded. (may or may not be specific to the right-end boundary.)

good: Multiple chromosomes of data are saved in a single file (which is the main point of the issue!).
good: The header correctly specifies the genome, not just the sequence (chromosome) name.

bad: Sometimes, an area of loaded data (a whole chromosome, the right side of a chromosome) is just not saved. ----> ACTUALLY... I think it is. Initially, I was testing this by looking at the file itself, and I assumed the last line for each chromosome represented the right-most saved region. But after I started dropping files in to IGB to see what they had, I never found incomplete files. I think I must have another track that was only loaded up to Chr1 8,277,465 and it was the last one written to the file. So there isn't any loaded data that is getting left out, its just that there is a lot of extra data included. ---> this is just as bad, but it is a different problem.

bad: IGB froze on me a couple times when I hit Save Track As .... that may be part of this issue, or it may be its own issue. --> this is difficult to reproduce.
bad: The default type is inconsistent (sometimes .bedgraph, sometimes .gr) -> that's another issue.
bad: The file extension has an extra ".*.bedgraph" on the end. -> that's another issue.

poor: the header has extra spaces (sometimes there is a " " after the "=" and sometimes not.) THIS part of the header could be separated into its own issue.

This sounds like a memory usage issue.
Check memory upon fail to write and freeze-up.
Also, use debugger to track what happens upon save.

Jenny issue a pull request and I made comments on it - see https://bitbucket.org/lorainelab/integrated-genome-browser/pull-requests/533/igbf-1090/diff

The save track feature is taking the header information from the wrong track.
In my last test it used the BA1 graph track settings to make the header, even when I had selected a different track and it wrote out the data based on the track I selected. When I deleted the BA1 track, the program froze.

I repeated this. BA1 was the first track loaded, and that was always the track used to make the header, regardless of which graph I selected to save the file.
When I deleted BA1, the header was based on BA2 (which was the second file loaded), when C1 was the track selected.
I hid the BA2 track, and tried to save a track from a the C1 graph. IGB froze.

In one test, IGB never froze. It consistently made the header for the bedgraph file from the first graph track in "the list", where "the list" is some structure that tracks are added to when data is first loaded (not when the track is added).
I added 6 graphs. I loaded data for each one individual (in a different order). I saved a series of bedraphs, and each time I deleted the track the header was made from. All tracks were removed in the order in which data was loaded.

Obviously, IGB is SUPPOSED to be making the header from the track that I selected.

It looks like IGB is actually saving ALL data from ALL graph tracks.

I added track BA1, loaded a section of data.
I added track BA2, loaded a different section of data for only that track.
I selected BA2 and saved. The saved file has data for both regions!!!! (the region that was only loaded for BA2 AND the region that was only loaded for BA1). The header is based on BA1.
See image SavedDataFromTwoSeparateTracks.png.

I was even able to save a single track with data from three separate tracks.
Further,
I found that where multiple tracks had data loaded, IGB did save the information from ALL of them to the output file.
So the bedgraph file ("T13.3") had a bunch of lines representing BA1, then a bunch of lines representing the SAME REGION with data from BA2.
I copied each of these into separate files and loaded them to confirm this.

In fact, when the graphs are in "line" mode, the track for file T13.3 clearly has overlapping data.
See image WhatsReallySaved.png

This represents a fundamental flaw in this feature.

Regardless of which track is selected, the output is identical.
The header is based on the first track that was loaded.
The data saved includes all data loaded from all tracks.

Example output file:

1. genome_version = A_thaliana_Jun_2009
   track type=bedgraph name= " BA1_2 5uM 10 day old seedlings Rep1, coverage" description= " BA1_2 5uM 10 day old seedlings Rep1, coverage" visibility=full color=204,0,204 viewLimits=0.0:1.0
   Chr1 8227365 8227465 1.0
   Chr1 8227365 8227465 1.0
   Chr1 8227365 8227465 1.0
   Chr1 8227465 8227503 0.0
   Chr1 8227503 8227509 1.0
   Chr1 8227509 8227553 2.0
   Chr1 8227509 8227553 2.0
   Chr1 8227503 8227509 1.0

Notice that "Chr1 8227509 8227553 2.0" (near the bottom) appears twice. The first time I loaded data, the edge fell in this range. After I zoomed out, the edge of the "new" range to download also fell in this range (this will be the case EVERY time that you load, zoom out, load more).

Notice that "8227365 8227465" is in there (at the top) three times. This range at the edge of the view when I loaded data, then I zoomed out just a bit, so the edge was still in this range. Everytime there is new range (any new range on the coordinate axis) data is loaded for that range. So the Same line of data from the graph file is added for every time that it is the edge of the new range.
bedgraph files should not have this duplication.

I need to check bedgraph criteria to see if the format requires non-overlap, but I'm pretty sure programs that use it will either mess up or spit out a file that has duplication like this.

Whatever function handles writing this file should make sure that there are no duplicated lines.

In the outputs that contain data from multiple files, it is clear that the output is not sorted.
Being sorted may not be a requirement of the file format, but it is a requirement for several programs that take in a bedgraph, and it makes life a LOT easier on anyone who has to look at it manually. This is also something that should be handled by whatever method writes writes this file type.

Testing IGB live:
v. 9.0.0

I set up to reproduce the line duplication. The first time I clicked Save Track As... IGB froze.
The second time it was fine.
(froze once more another time in this testing, when I clicked Save Track As... but I still don't know what causes it)

-IGB live does produce the duplicated lines. (see * below) --> IGBF-1147 better bedgraph formatting
-The lines are not (strictly) ordered. (See **) --> IGBF-1147 better bedgraph formatting
-The header does not have extra spaces after '='.
-The header is respected: when the saved track is loaded it shows the name from the header, not the file name. IGB also displays the file with -the color and y-axis range of the original track. In testing the branch, only the y-axis range was respected from the header. I think this is because of the header formatting because when I dropped a file made by IGB live into the window for IGB 1090, the header was fully respected.
-The header is made from the correct tier.
-The default file type (.gr or .bedgraph) is inconsistent.
-Only data from the selected track is saved.

IGB 1090 bedgraph header:

1. genome_version = A_thaliana_Jun_2009
   track type=bedgraph name= " BA1_2 5uM 10 day old seedlings Rep1, coverage" description= " BA1_2 5uM 10 day old seedlings Rep1, coverage" visibility=full color=204,0,204 viewLimits=0.0:1.0

IGB live bedgraph header:

1. genome_version = Chr1
   track type=bedgraph name=" C2 DMSO only 10 day old seedling Rep2, coverage" description=" C2 DMSO only 10 day old seedling Rep2, coverage" visibility=full color=0,128,0 viewLimits=0.0:3.0

Chr1 8227506 8227522 2.0
Chr1 8227506 8227522 2.0 *
Chr1 8227534 8227542 5.0
Chr1 8227534 8227542 5.0
Chr1 8227542 8227553 6.0
Chr1 8227534 8227542 5.0
Chr1 20998229 20998235 5.0
Chr1 20998235 20998333 0.0
Chr1 8227506 8227522 2.0 **

Quick check on the annotation saving in IGB live:

bed files are saved as name.bed.*.bed. ---so that is shared
Multiple chromosomes of data are saved, and only data from the selected track is saved. --so the annotation saving stuff is probably a good thing to copy-cat from.
In bed detail, each line ends with, N/A N/A

The bit about the header not being respected is due to the spaces.
See Wiggle.java

513 bw.write("track type=" + getTrackType() + " name= \"" + gname + "\"");
514 bw.write(" description= \"" + human_name + "\"");

They were added in this branch and they need to be taken back out.

good: Multiple chromosomes of data are saved in a single file (which is the main point of the issue!).
good: The header correctly specifies the genome, not just the sequence (chromosome) name.

bad: The header has extra spaces (sometimes there is a " " after the "=" ). This was introduced on this branch. (See last comment)
bad: All loaded data (from all graph tracks) is being exported.
bad: The information for the header is taken from the wrong tier.

The above problems are all introduced in this branch.
We need to really re-think how we are getting it to save multiple chromosomes of data.
Somehow annotation data is saved from all chromosomes while still only taking data from the desired track.

Started a new branch and copied the changes from the original onto this new branch in one, clean commit. Link to new branch:

https://bitbucket.org/jdaly8/integrated-genome-browser/branch/IGBF-1090-b

The latest iteration of this branch is on Ivory's remote branch:

https://bitbucket.org/IvoryBlak/integrated-genome-browser/branch/IGBF-1090-b

Code review tip:
--the code was simplified and commented in the most recent commit.

Functionality review tips:
--Load data from multiple data sets, and on multiple chromosomes. Make sure the saved file reflects ONLY the selected track and has data for ALL chromosomes where data was loaded.
--When you save a file, you can drag/drop the new file onto IGB to quickly see which regions were saved. Files saved in by IGB should load correctly in IGB.
--Test the above for both annotation tracks and graphs tracks. (for everything else, you can just check graph tracks)
--Graph tracks saved in IGB should have the same name and color scheme as the original file (only after loading data from the saved file).
--Look at the saved graph file in a text editor. The genome in the header should match the GENOME not the chromosome.

This issue is ready for review.

#1 1st level review

• Color, name, description match selected track
• Only selected track saves, for all loaded chromosomes
• Annotations working same as before, doesn't have a header and still saves multiple chromosomes.

#2 1st-Level Review:

• Only the selected track saves for any chromosome that has been loaded.
• Annotation tracks are being saved as expected with no header
• Graph tracks have the same track name and color scheme as the original file

Since this issue has been reviewed twice and passed the reviews, I am moving the issue to the Needs Pull Request Column and assigning the issue to Ivory.

While testing this story on the development branch I noticed that it was not functioning as expected because data was not being saved for ALL chromosomes in which data had been loaded. Is it possible that the code that made it to eos.transvar.org comes from Ivory's story IGBF-1090 instead of her working branch called IGBF-1090-b?

To reproduce:

1) Download and run IGB from http://eos.transvar.org/igb/
2) Open A. thaliana genome and load some bedgraph data on Chr1 and Chr3
3) Now Save the bedgraph track as a new file (do this from Chr3)
4) Open th saved track in IGB and load data for Chr 1 and Chr 3.

*Observe: There is no data on Chr 1 for the new track, but there should be.

I am re-assinging this to Ivory and moving it back to the Reviewing Pull-Request column.

I just updated my master branch and tested. Like Mason said, data from a graph track is saved for ONLY the current chromosome, not for all loaded chromosomes.

The most recent changes on branch IGBF-1090-b are not reflected in the current master branch.
I searched in the commits for main master and I do not see any commit where branch IGBF-1090 or IGBF-1090-b was ever merged in.

If you can give me the commit id or pull request number so I can see exactly what was merged in I can look into this further.
My current best guess is that this branch never was merged into master.

I rebased branch IGBF-1090-b from master, tested (still good) and submitted a pull request.

After testing, it seems the issue is resolved.

*Only the selected track saves for any chromosome that has been loaded.
*Annotation tracks are being saved as expected with no header
*Graph tracks have the same track name and color scheme as the original file
*I could not find any other side effects or issues related to this change.

Since this issue is resolved, it will now be closed.