Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-3450

Run nf-core/rnaseq pipeline using SL4 target genome assembly

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      Goal:

      Create BAM, coverage graph, and junction files for the Muday Lab time course data using SL4 genome and gene annotations as the target for alignment.

      When complete, please note the location of files created.

      Note: Don't use the renamed files. Use the original input files prior to their being renamed.

        Attachments

          Activity

          ann.loraine Ann Loraine created issue -
          ann.loraine Ann Loraine made changes -
          Field Original Value New Value
          Epic Link IGBF-3446 [ 22548 ]
          Mdavis4290 Molly Davis made changes -
          Sprint Fall 2 2023 Sep 17 [ 178 ]
          Hide
          Mdavis4290 Molly Davis added a comment - - edited

          Directory of original files that are NOT renamed: /projects/tomato_genome/rnaseq/muday144-timeSeries-checkReadMEFIRST/00_fastq

          SL4 Nf-core run location: /projects/tomato_genome/fnb/dataprocessing/muday-144/nfcore-SL4

          Show
          Mdavis4290 Molly Davis added a comment - - edited Directory of original files that are NOT renamed : /projects/tomato_genome/rnaseq/muday144-timeSeries-checkReadMEFIRST/00_fastq SL4 Nf-core run location : /projects/tomato_genome/fnb/dataprocessing/muday-144/nfcore-SL4
          Mdavis4290 Molly Davis made changes -
          Status To-Do [ 10305 ] In Progress [ 3 ]
          ann.loraine Ann Loraine made changes -
          Description Goal:

          Create BAM, coverage graph, and junction files for the Muday Lab time course data using SL4 genome and gene annotations as the target for alignment.

          When complete, please note the location of files created.




          Goal:

          Create BAM, coverage graph, and junction files for the Muday Lab time course data using SL4 genome and gene annotations as the target for alignment.

          When complete, please note the location of files created.

          Note: Don't use the renamed files. Use the original input files prior to their being renamed.



          Hide
          Mdavis4290 Molly Davis added a comment - - edited

          Nextflow Pipeline ran successfully with SL4 genome
          Directory: /projects/tomato_genome/fnb/dataprocessing/muday-144/nfcore-SL4
          MultiQC report notes: No errors or warnings were present in the report. The output file is named 'muday-144-SL4_multiqc_report.html'.

          Next Steps: Commit Multiqc report to documentation in flavonoid repo. Create Coverage graphs and junction files.

          Show
          Mdavis4290 Molly Davis added a comment - - edited Nextflow Pipeline ran successfully with SL4 genome Directory : /projects/tomato_genome/fnb/dataprocessing/muday-144/nfcore-SL4 MultiQC report notes : No errors or warnings were present in the report. The output file is named 'muday-144-SL4_multiqc_report.html'. Next Steps : Commit Multiqc report to documentation in flavonoid repo. Create Coverage graphs and junction files.
          Hide
          Mdavis4290 Molly Davis added a comment - - edited

          Launch renameBams.sh script:
          ./renameBams.sh
          Launch Scaled Coverage graphs script:
          ./sbatch-doIt.sh .bam bamCoverage.sh >jobs.out 2>jobs.err
          Launch Junction files script:
          ./sbatch-doIt.sh .bam find_junctions.sh >jobs.out 2>jobs.err

          Reviewer:
          Check that files have reasonable sizes (no "zero" size files, for example)
          Check that every "FJ.bed.gz" file has a corresponding "FJ.bed.gz.tbi" index file
          Check that every bam file has a corresponding "FJ.bed.gz" file
          Check that every bam file has a corresponding "scaled.bedgraph.gz" file
          Check that every "scaled.bedgraph.gz" has a corresponding "scaled.bedgraph.gz.tbi"

          Reviewer: [~RobertReid]

          Note: There were two warnings but no errors that I saw.

          Show
          Mdavis4290 Molly Davis added a comment - - edited Launch renameBams.sh script : ./renameBams.sh Launch Scaled Coverage graphs script : ./sbatch-doIt.sh .bam bamCoverage.sh >jobs.out 2>jobs.err Launch Junction files script : ./sbatch-doIt.sh .bam find_junctions.sh >jobs.out 2>jobs.err Reviewer : Check that files have reasonable sizes (no "zero" size files, for example) Check that every "FJ.bed.gz" file has a corresponding "FJ.bed.gz.tbi" index file Check that every bam file has a corresponding "FJ.bed.gz" file Check that every bam file has a corresponding "scaled.bedgraph.gz" file Check that every "scaled.bedgraph.gz" has a corresponding "scaled.bedgraph.gz.tbi" Reviewer: [~RobertReid] Note: There were two warnings but no errors that I saw.
          Hide
          Mdavis4290 Molly Davis added a comment - - edited

          Branch: https://bitbucket.org/mdavis4290/molly3-flavonoid-rnaseq/branch/IGBF-3450

          Note: Includes the SL4 Mutliqc report.

          [~aloraine]

          Show
          Mdavis4290 Molly Davis added a comment - - edited Branch : https://bitbucket.org/mdavis4290/molly3-flavonoid-rnaseq/branch/IGBF-3450 Note: Includes the SL4 Mutliqc report. [~aloraine]
          Mdavis4290 Molly Davis made changes -
          Assignee Molly Davis [ molly ]
          Mdavis4290 Molly Davis made changes -
          Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
          ann.loraine Ann Loraine made changes -
          Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
          ann.loraine Ann Loraine made changes -
          Assignee Ann Loraine [ aloraine ]
          Hide
          ann.loraine Ann Loraine added a comment - - edited

          Change requests:

          1) "chmod" the entire contents of nfcore-SL4 to make them group editable

          This is so that I can move the bam, bedgraph, and junction files over to another directory for copying to the data host. I don't want to make copies due to the risk of error and also to save disk space and energy.

          2) Add this file to your branch:

          • /projects/tomato_genome/fnb/dataprocessing/muday-144/nfcoreSL4/results/star_salmon/salmon.merged.gene_counts.tsv

          as file:

          • 72_F3H_PollenTube/muday-144-SL4_salmon.merged.gene_counts.tsv

          Please add it to your branch and then submit a PR.

          Thank you!

          Ann

          Show
          ann.loraine Ann Loraine added a comment - - edited Change requests: 1) "chmod" the entire contents of nfcore-SL4 to make them group editable This is so that I can move the bam, bedgraph, and junction files over to another directory for copying to the data host. I don't want to make copies due to the risk of error and also to save disk space and energy. 2) Add this file to your branch: /projects/tomato_genome/fnb/dataprocessing/muday-144/nfcoreSL4/results/star_salmon/salmon.merged.gene_counts.tsv as file: 72_F3H_PollenTube/muday-144-SL4_salmon.merged.gene_counts.tsv Please add it to your branch and then submit a PR. Thank you! Ann
          Hide
          ann.loraine Ann Loraine added a comment - - edited

          Double-checking:

          The renamed fastq files were NOT used here. Is this right? That is, the original files with their original names were processed, the same exact files that we got from the sequencer?

          Show
          ann.loraine Ann Loraine added a comment - - edited Double-checking: The renamed fastq files were NOT used here. Is this right? That is, the original files with their original names were processed, the same exact files that we got from the sequencer?
          ann.loraine Ann Loraine made changes -
          Status First Level Review in Progress [ 10301 ] To-Do [ 10305 ]
          ann.loraine Ann Loraine made changes -
          Rank Ranked higher
          ann.loraine Ann Loraine made changes -
          Assignee Ann Loraine [ aloraine ] Molly Davis [ molly ]
          Hide
          Mdavis4290 Molly Davis added a comment - - edited

          These are the original files that are NOT renamed. Straight from the sequencer!
          This can be confirmed in the README in the original directory: /projects/tomato_genome/rnaseq/muday144-timeSeries-checkReadMEFIRST/00_fastq

          Results should also now be editable.
          [~aloraine]

          Show
          Mdavis4290 Molly Davis added a comment - - edited These are the original files that are NOT renamed. Straight from the sequencer! This can be confirmed in the README in the original directory: /projects/tomato_genome/rnaseq/muday144-timeSeries-checkReadMEFIRST/00_fastq Results should also now be editable. [~aloraine]
          Mdavis4290 Molly Davis made changes -
          Status To-Do [ 10305 ] In Progress [ 3 ]
          Show
          Mdavis4290 Molly Davis added a comment - Pull Request : https://bitbucket.org/hotpollen/flavonoid-rnaseq/pull-requests/24
          Mdavis4290 Molly Davis made changes -
          Assignee Molly Davis [ molly ]
          Mdavis4290 Molly Davis made changes -
          Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
          Mdavis4290 Molly Davis made changes -
          Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
          Mdavis4290 Molly Davis made changes -
          Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
          Mdavis4290 Molly Davis made changes -
          Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
          Mdavis4290 Molly Davis made changes -
          Assignee Ann Loraine [ aloraine ]
          Hide
          ann.loraine Ann Loraine added a comment -

          Testing suggestions:

          • Compare multiqc reports from SL4, SL5 assemblies. PCA plots should mostly match.
          Show
          ann.loraine Ann Loraine added a comment - Testing suggestions: Compare multiqc reports from SL4, SL5 assemblies. PCA plots should mostly match.
          Hide
          ann.loraine Ann Loraine added a comment -

          [~molly]: please try again with the "chmod-ing"

          I need to move these files into position for transferring to data.bioviz.org, like so:

          [aloraine@str-i1 muday-2022-timeseries]$ pwd
          /projects/tomato_genome/fnb/dataprocessing/muday-144/for_quickload/S_lycopersicum_Sep_2019/muday-2022-timeseries
          [aloraine@str-i1 muday-2022-timeseries]$ mv ../../../nfcore-SL4/results/star_salmon/*.bam .
          mv: cannot move '../../../nfcore-SL4/results/star_salmon/7-are-15-min-28C.bam' to './7-are-15-min-28C.bam': Permission denied
          

          I need to move all the bam, bam index, coverage graph, coverage graph index, junction, junction index files over to /projects/tomato_genome/fnb/dataprocessing/muday-144/for_quickload/S_lycopersicum_Sep_2019/muday-2022-timeseries but can't because of how the permissions are set.

          I would like to take some pictures of genes before tomorrow if at all possible! But I don't want to waste a lot of disk space and energy to meet a deadline

          Show
          ann.loraine Ann Loraine added a comment - [~molly] : please try again with the "chmod-ing" I need to move these files into position for transferring to data.bioviz.org, like so: [aloraine@str-i1 muday-2022-timeseries]$ pwd /projects/tomato_genome/fnb/dataprocessing/muday-144/for_quickload/S_lycopersicum_Sep_2019/muday-2022-timeseries [aloraine@str-i1 muday-2022-timeseries]$ mv ../../../nfcore-SL4/results/star_salmon/*.bam . mv: cannot move '../../../nfcore-SL4/results/star_salmon/7-are-15-min-28C.bam' to './7-are-15-min-28C.bam': Permission denied I need to move all the bam, bam index, coverage graph, coverage graph index, junction, junction index files over to /projects/tomato_genome/fnb/dataprocessing/muday-144/for_quickload/S_lycopersicum_Sep_2019/muday-2022-timeseries but can't because of how the permissions are set. I would like to take some pictures of genes before tomorrow if at all possible! But I don't want to waste a lot of disk space and energy to meet a deadline
          ann.loraine Ann Loraine made changes -
          Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
          ann.loraine Ann Loraine made changes -
          Status Reviewing Pull Request [ 10303 ] To-Do [ 10305 ]
          ann.loraine Ann Loraine made changes -
          Assignee Ann Loraine [ aloraine ] Molly Davis [ molly ]
          Mdavis4290 Molly Davis made changes -
          Assignee Molly Davis [ molly ] Ann Loraine [ aloraine ]
          Hide
          Mdavis4290 Molly Davis added a comment - - edited

          chmod g+w /projects/tomato_genome/fnb/dataprocessing/muday-144/nfcore-SL4/results
          chmod -R g+w *

          Show
          Mdavis4290 Molly Davis added a comment - - edited chmod g+w /projects/tomato_genome/fnb/dataprocessing/muday-144/nfcore-SL4/results chmod -R g+w *
          Mdavis4290 Molly Davis made changes -
          Status To-Do [ 10305 ] In Progress [ 3 ]
          Mdavis4290 Molly Davis made changes -
          Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
          Hide
          ann.loraine Ann Loraine added a comment -

          Moved or copied nf-core output files to HPC file system location: /projects/tomato_genome/fnb/dataprocessing/muday-144/for_quickload/S_lycopersicum_Sep_2019/muday-2022-timeseries

          On data.bioviz.org host, launched "screen" session and commenced copying nf-core output files to data.bioviz.org with:

          cd /mnt/igbdata/hotpollen/S_lycopersicum_Sep_2019
          rsync -rz aloraine@10.16.115.244:/projects/tomato_genome/fnb/dataprocessing/muday-144/for_quickload/S_lycopersicum_Sep_2019/muday-2022-timeseries .
          
          Show
          ann.loraine Ann Loraine added a comment - Moved or copied nf-core output files to HPC file system location: /projects/tomato_genome/fnb/dataprocessing/muday-144/for_quickload/S_lycopersicum_Sep_2019/muday-2022-timeseries On data.bioviz.org host, launched "screen" session and commenced copying nf-core output files to data.bioviz.org with: cd /mnt/igbdata/hotpollen/S_lycopersicum_Sep_2019 rsync -rz aloraine@10.16.115.244:/projects/tomato_genome/fnb/dataprocessing/muday-144/for_quickload/S_lycopersicum_Sep_2019/muday-2022-timeseries .
          Hide
          ann.loraine Ann Loraine added a comment -

          Added new SL4 data to the annots.xml configuration files for visualization in IGB. Updated "main" branch of https://bitbucket.org/hotpollen/genome-browser-visualization/src/main/

          Show
          ann.loraine Ann Loraine added a comment - Added new SL4 data to the annots.xml configuration files for visualization in IGB. Updated "main" branch of https://bitbucket.org/hotpollen/genome-browser-visualization/src/main/
          ann.loraine Ann Loraine made changes -
          Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
          Hide
          ann.loraine Ann Loraine added a comment -

          Review passes:

          • counts file is added to the repo
          • data files are added to data.bioviz.org for visualization and permissions set to allow delivery via the Web server
          • also added multiqc report and its input data to the data.bioviz.org deployment site for easier visualization

          Note that these data are NOT using renamed labels. The files that were processed match the original file names from the sequencer.

          Moving to ready for testing.

          Show
          ann.loraine Ann Loraine added a comment - Review passes: counts file is added to the repo data files are added to data.bioviz.org for visualization and permissions set to allow delivery via the Web server also added multiqc report and its input data to the data.bioviz.org deployment site for easier visualization Note that these data are NOT using renamed labels. The files that were processed match the original file names from the sequencer. Moving to ready for testing.
          ann.loraine Ann Loraine made changes -
          Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
          ann.loraine Ann Loraine made changes -
          Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
          ann.loraine Ann Loraine made changes -
          Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
          ann.loraine Ann Loraine made changes -
          Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
          ann.loraine Ann Loraine made changes -
          Assignee Ann Loraine [ aloraine ]
          Mdavis4290 Molly Davis made changes -
          Assignee Molly Davis [ molly ]
          Mdavis4290 Molly Davis made changes -
          Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
          Hide
          Mdavis4290 Molly Davis added a comment - - edited

          Testing:

          Moving to done!

          Show
          Mdavis4290 Molly Davis added a comment - - edited Testing : Count file is present in repo: 72_F3H_PollenTube/muday-144-SL4_salmon.merged.gene_counts.tsv SL5 (PT time GM lab) and SL4 (PT time GM lab) datasets for Muday-144 are present and accessible in IGB quickload Found report and input data on deployment site here: https://data.bioviz.org/hotpollen/S_lycopersicum_Sep_2019/muday-2022-timeseries/ Moving to done!
          Mdavis4290 Molly Davis made changes -
          Resolution Done [ 10000 ]
          Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]

            People

            • Assignee:
              Mdavis4290 Molly Davis
              Reporter:
              ann.loraine Ann Loraine
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: