The "data processing" directory on the cluster can be found here: /projects/tomato_genome/fnb/dataprocessing
Most scripts can be found here: /projects/tomato_genome/fnb/scripts/flavonoid-rnaseq/src

Here is the entire workflow for processing the tardigrade data:

1. Run prefetch.sh to get .sra files for the list of SRR accessions.
2. Check that the .sra files got downloaded correctly.
3. Run fasterq-dump to make fastq files from the .sra files.
4. Compress the fastq files with gzip using gzip.sh and submit to slurm queue using sbatch-doIt.sh
5. Run nf-core/rnaseq pipeline latest version. For this, you have to make a "parameter" file and provide gene annotations (.gtf file), genome annotations (.bed file), and genome assembly (.fasta file).
6. Check that the pipeline ran correctly.
7. Rename the BAM files that the pipeline made using rename.sh
8. Make a folder with links to the bam files and their indexes. Make coverage graphs with bamCoverage.sh by submitting to the slurm cluster using sbatch-doIt.sh.
9. Make junction files with find_junctions.sh (located here: /projects/tomato_genome/fnb/alt_splicing/SRP371294/results/star_salmon) by submitting to the slurm cluster using sbatch-doIt.sh.
10. Check that steps 8 and 9 ran correctly.
11. Make a folder called "for_quickload" as a direct child folder in each "SRP" folder. Make a directory named for the genome assembly, same name as used in IGB. Inside there, make another folder named for the SRP accession. We will use this file subtree to copy the data files to a new host using rsync.
12. Move (don't copy) bam, bam.bai, coverage graph files, and junction files into the above child folder.
13. Make sure that all the files everywhere are group-readable, group-writeable. Make sure that all the folders (directories) are group-readable, group-executable, and group-writeable.

Based on the mastersheet in the linked ticket, there should be 10 "SRP" folders at the end of this process (11 if we're including the experiment with only an ERR accession).

I'm running this pipeline with the smallest dataset from the mastersheet (ERP134165, six samples) so that I can optimize it before running all of the tardigrade data through it.

1. The prefetch script ran without issue. The .sra files can be found in their respective SRR-numbered folders in /projects/tomato_genome/fnb/dataprocessing/tardigrade/ERP134165/
2. I've confirmed that the .sra files got downloaded correctly by 1) ensuring that vdb-validate finished running with each .sra file without any exit codes (i.e., checking .out files), 2) ensuring that there are no error messages present in the .err files, and 3) ensuring that .sra files are present in each SRR-numbered folder and are of a non-zero size.

3. The fasterq-dump script did not run successfully. In each of the .out files produced from this script, there was an error message that the projects folder had hit its capacity for disk space.

This was happening because the script was copying fastq files from one location to another, therefore creating two copies of the same fastq file each time it's run (and these files are not small!).

To fix this issue, the script has been modified such that fastq files are only ever moved, not copied, and are gzipped to further reduce the amount of space they take up on the cluster. This makes step 4 of the pipeline above redundant, so it should be removed from future versions.

The fastq files output from the fasterq-dump script are located here: /projects/tomato_genome/fnb/dataprocessing/tardigrade/ERP134165/fastq/
The fastqc reports produced and used to confirm that the fastq files downloaded properly are located here: /projects/tomato_genome/fnb/dataprocessing/tardigrade/ERP134165/fastq/fastqc/

For review:

• Navigate to: /projects/tomato_genome/fnb/dataprocessing/tardigrade/ERP134165/
• Have I set up the "SRP folder" and all of its subfolders with the correct permissions thus far?
• Are the names of these subfolders logical such that they clue you in to what they contain?
• Are there folders for each SRR number/sample?
• Is there an .sra file in each of those folders that corresponds to that sample?
• Navigate to: /projects/tomato_genome/fnb/dataprocessing/tardigrade/ERP134165/fastq/
• Is there a .fastq file downloaded for each sample?

Please see the linked ticket for documentation of the steps I took to produce this data.

Everything looks OK, except that the symbolic links in /projects/tomato_genome/fnb/dataprocessing/tardigrade/ERP134165 are flashing red. I don't think this is a problem, however.
Moving to Done.