Details
-
Type:
Task
-
Status: Closed (View Workflow)
-
Priority:
Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:None
-
Story Points:2
-
Epic Link:
-
Sprint:Fall 3 2021 Sep 13 - Sep 24, Fall 4 2021 Sep 27 - Oct 8
Description
Nextflow provides a standard type of RNA-Seq data pipeline called "nf-core/rnaseq" that we can potentially use for processing data in the Pollen PGRP and other related projects.
For example, the following command shows an example of running this pipeline on a "single end" data set, using a reference genome indicated by the "--fasta" option:
nextflow run nf-core/rnaseq -profile conda --singleEnd --reverseStranded --skipTrimming --reads '*.fastq.gz' --fasta 'ftp://ftp.ensembl.org/pub/release-99/fasta/rattus_norvegicus/dna_index/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa.gz' --gtf 'ftp://ftp.ensembl.org/pub/release-99/gtf/rattus_norvegicus/Rattus_norvegicus.Rnor_6.0.99.gtf.gz' --fc_count_type transcript
One major benefit of using this pipeline is that there is a lot of support for it in the larger bioinformatics community. There's a company (Sequera) that is supporting this and other Nextflow related projects.
Attachments
Issue Links
Activity
| Field | Original Value | New Value |
|---|---|---|
| Epic Link | IGBF-2323 [ 18477 ] |
| Description |
{code} nextflow run nf-core/rnaseq -profile conda --singleEnd --reverseStranded --skipTrimming --reads '*.fastq.gz' --fasta 'ftp://ftp.ensembl.org/pub/release-99/fasta/rattus_norvegicus/dna_index/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa.gz' --gtf 'ftp://ftp.ensembl.org/pub/release-99/gtf/rattus_norvegicus/Rattus_norvegicus.Rnor_6.0.99.gtf.gz' --fc_count_type transcript {code} |
Nextflow provides a standard type of RNA-Seq data pipeline called "nf-core/rnaseq" that we can potentially use for processing data in the Pollen PGRP and other related projects.
For example, the following command shows ab example of running this pipeline on a "single end" data set, using a reference genome indicated by the "--fasta" option: {code} nextflow run nf-core/rnaseq -profile conda --singleEnd --reverseStranded --skipTrimming --reads '*.fastq.gz' --fasta 'ftp://ftp.ensembl.org/pub/release-99/fasta/rattus_norvegicus/dna_index/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa.gz' --gtf 'ftp://ftp.ensembl.org/pub/release-99/gtf/rattus_norvegicus/Rattus_norvegicus.Rnor_6.0.99.gtf.gz' --fc_count_type transcript {code} One major benefit of using this pipeline is that there is a lot of support for it in the larger bioinformatics community. There's a company (Sequera) that is supporting this and other Nextflow related projects, and |
| Status | To-Do [ 10305 ] | In Progress [ 3 ] |
| Assignee | Ann Loraine [ aloraine ] |
Dealing with lack of internet access on nodes:
https://nf-co.re/tools/#downloading-pipelines-for-offline-use
Ran the test code on the head node, but it failed on a cluster node because the cluster node can't access the internet. This means I have to install nf-core tools and download all the things in advance.
Test command:
nextflow run nf-core/rnaseq -profile test,singularity
Requested nf-core to be installed on the HPC system. Once this is installed, it will be easier to manage import of digital artifacts needed for running jobs.
nf-core was installed. I used it to download all singularity images, configs, and code needed to run the rnaseq nf-core pipeline to: /nobackup/tomato_genome/nfcore with:
nf-core download rnaseq
after running:
module load nf-core
which launched a virtual environment with nf-core commands and python configured.
Following directions on off-line use at https://nf-co.re/usage/offline, ran:
nextflow run nf-core-rnaseq-3.3/workflow -profile test,singularity
Unfortunately, the test profile only works if you have internet access on the executing node. Was able to run it on the head node, however.
| Sprint | Fall 3 2021 Sep 13 - Sep 24 [ 129 ] | Fall 3 2021 Sep 13 - Sep 24, Fall 4 2021 Sep 27 - Oct 8 [ 129, 130 ] |
| Rank | Ranked higher |
Warning obtained on running rnaseq pipeline:
WARN: ============================================================================= Biotype attribute 'gene_biotype' not found in the last column of the GTF file! Biotype QC will be skipped to circumvent the issue below: https://github.com/nf-core/rnaseq/issues/460 Amend '--featurecounts_group_type' to change this behaviour. ===================================================================================
used script:
#!/bin/env bash SAMPLESHEET=$1 GENOMEFASTA=$2 GTF=$3 GENEBED=$4 MS="Must provide samples, reference genome fasta, annotations GTF, and annotations BED files" EX="Example: ./doIt.sh samples.csv S_lycopersicum_Sep_2019.fa S_lycopersium_Sep_2019.gtf S_lycopersium_Sep_2019.bed" if [[ -z "$SAMPLESHEET" ]]; then echo "$MS" 1>&2 echo "$EX" 1>&2 exit 1 fi if [[ -z "$GENOMEFASTA" ]]; then echo "$MS" 1>&2 echo "$EX" 1>&2 exit 1 fi if [[ -z "$GTF" ]]; then echo "$MS" 1>&2 echo "$EX" 1>&2 exit 1 fi if [[ -z "$GENEBED" ]]; then echo "$MS" 1>&2 echo "$EX" 1>&2 exit 1 fi # for running on cluster nodes lacking internet connection, download the pipeline # and Singularity containers as directed in https://nf-co.re/usage/offline nextflow run nf-core-rnaseq-3.3/workflow -resume -profile singularity --fasta $GENOMEFASTA --input $SAMPLESHEET --gtf $GTF --gene_bed $GENEBED
Pipeline ran with:
doIt.sh 1> out 2> err
Stdout is attached.
| Attachment | 2021-09-28-rnaseq_pipeline_stdout.txt [ 16786 ] |
| Status | In Progress [ 3 ] | Needs 1st Level Review [ 10005 ] |
| Assignee | Ann Loraine [ aloraine ] |
Samples file used:
sample,fastq_1,fastq_2,strandedness are-120min-28-34C-R1,are-120min-28-34C-R1_R1_001.fastq.gz,are-120min-28-34C-R1_R2_001.fastq.gz,reverse are-120min-28-34C-R2,are-120min-28-34C-R2_R1_001.fastq.gz,are-120min-28-34C-R2_R2_001.fastq.gz,reverse are-120min-28-34C-R3,are-120min-28-34C-R3_R1_001.fastq.gz,are-120min-28-34C-R3_R2_001.fastq.gz,reverse are-120min-28C-R1,are-120min-28C-R1_R1_001.fastq.gz,are-120min-28C-R1_R2_001.fastq.gz,reverse are-120min-28C-R2,are-120min-28C-R2_R1_001.fastq.gz,are-120min-28C-R2_R2_001.fastq.gz,reverse are-120min-28C-R3,are-120min-28C-R3_R1_001.fastq.gz,are-120min-28C-R3_R2_001.fastq.gz,reverse F3H-OX3-120min-28-34C-R1,F3H-OX3-120min-28-34C-R1_R1_001.fastq.gz,F3H-OX3-120min-28-34C-R1_R2_001.fastq.gz,reverse F3H-OX3-120min-28-34C-R2,F3H-OX3-120min-28-34C-R2_R1_001.fastq.gz,F3H-OX3-120min-28-34C-R2_R2_001.fastq.gz,reverse F3H-OX3-120min-28-34C-R3,F3H-OX3-120min-28-34C-R3_R1_001.fastq.gz,F3H-OX3-120min-28-34C-R3_R2_001.fastq.gz,reverse F3H-OX3-120min-28C-R1,F3H-OX3-120min-28C-R1_R1_001.fastq.gz,F3H-OX3-120min-28C-R1_R2_001.fastq.gz,reverse F3H-OX3-120min-28C-R2,F3H-OX3-120min-28C-R2_R1_001.fastq.gz,F3H-OX3-120min-28C-R2_R2_001.fastq.gz,reverse F3H-OX3-120min-28C-R3,F3H-OX3-120min-28C-R3_R1_001.fastq.gz,F3H-OX3-120min-28C-R3_R2_001.fastq.gz,reverse F3H-OX4-120min-28-34C-R1,F3H-OX4-120min-28-34C-R1_R1_001.fastq.gz,F3H-OX4-120min-28-34C-R1_R2_001.fastq.gz,reverse F3H-OX4-120min-28-34C-R2,F3H-OX4-120min-28-34C-R2_R1_001.fastq.gz,F3H-OX4-120min-28-34C-R2_R2_001.fastq.gz,reverse F3H-OX4-120min-28-34C-R3,F3H-OX4-120min-28-34C-R3_R1_001.fastq.gz,F3H-OX4-120min-28-34C-R3_R2_001.fastq.gz,reverse F3H-OX4-120min-28C-R1,F3H-OX4-120min-28C-R1_R1_001.fastq.gz,F3H-OX4-120min-28C-R1_R2_001.fastq.gz,reverse F3H-OX4-120min-28C-R2,F3H-OX4-120min-28C-R2_R1_001.fastq.gz,F3H-OX4-120min-28C-R2_R2_001.fastq.gz,reverse F3H-OX4-120min-28C-R3,F3H-OX4-120min-28C-R3_R1_001.fastq.gz,F3H-OX4-120min-28C-R3_R2_001.fastq.gz,reverse VF36-120min-28-34C-R1,VF36-120min-28-34C-R1_R1_001.fastq.gz,VF36-120min-28-34C-R1_R2_001.fastq.gz,reverse VF36-120min-28-34C-R2,VF36-120min-28-34C-R2_R1_001.fastq.gz,VF36-120min-28-34C-R2_R2_001.fastq.gz,reverse VF36-120min-28-34C-R3,VF36-120min-28-34C-R3_R1_001.fastq.gz,VF36-120min-28-34C-R3_R2_001.fastq.gz,reverse VF36-120min-28C-R1,VF36-120min-28C-R1_R1_001.fastq.gz,VF36-120min-28C-R1_R2_001.fastq.gz,reverse VF36-120min-28C-R2,VF36-120min-28C-R2_R1_001.fastq.gz,VF36-120min-28C-R2_R2_001.fastq.gz,reverse VF36-120min-28C-R3,VF36-120min-28C-R3_R1_001.fastq.gz,VF36-120min-28C-R3_R2_001.fastq.gz,reverse
| Status | Needs 1st Level Review [ 10005 ] | First Level Review in Progress [ 10301 ] |
| Assignee | Nowlan Freese [ nfreese ] |
| Description |
Nextflow provides a standard type of RNA-Seq data pipeline called "nf-core/rnaseq" that we can potentially use for processing data in the Pollen PGRP and other related projects.
For example, the following command shows ab example of running this pipeline on a "single end" data set, using a reference genome indicated by the "--fasta" option: {code} nextflow run nf-core/rnaseq -profile conda --singleEnd --reverseStranded --skipTrimming --reads '*.fastq.gz' --fasta 'ftp://ftp.ensembl.org/pub/release-99/fasta/rattus_norvegicus/dna_index/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa.gz' --gtf 'ftp://ftp.ensembl.org/pub/release-99/gtf/rattus_norvegicus/Rattus_norvegicus.Rnor_6.0.99.gtf.gz' --fc_count_type transcript {code} One major benefit of using this pipeline is that there is a lot of support for it in the larger bioinformatics community. There's a company (Sequera) that is supporting this and other Nextflow related projects, and |
Nextflow provides a standard type of RNA-Seq data pipeline called "nf-core/rnaseq" that we can potentially use for processing data in the Pollen PGRP and other related projects.
For example, the following command shows an example of running this pipeline on a "single end" data set, using a reference genome indicated by the "--fasta" option: {code} nextflow run nf-core/rnaseq -profile conda --singleEnd --reverseStranded --skipTrimming --reads '*.fastq.gz' --fasta 'ftp://ftp.ensembl.org/pub/release-99/fasta/rattus_norvegicus/dna_index/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa.gz' --gtf 'ftp://ftp.ensembl.org/pub/release-99/gtf/rattus_norvegicus/Rattus_norvegicus.Rnor_6.0.99.gtf.gz' --fc_count_type transcript {code} One major benefit of using this pipeline is that there is a lot of support for it in the larger bioinformatics community. There's a company (Sequera) that is supporting this and other Nextflow related projects, and |
| Description |
Nextflow provides a standard type of RNA-Seq data pipeline called "nf-core/rnaseq" that we can potentially use for processing data in the Pollen PGRP and other related projects.
For example, the following command shows an example of running this pipeline on a "single end" data set, using a reference genome indicated by the "--fasta" option: {code} nextflow run nf-core/rnaseq -profile conda --singleEnd --reverseStranded --skipTrimming --reads '*.fastq.gz' --fasta 'ftp://ftp.ensembl.org/pub/release-99/fasta/rattus_norvegicus/dna_index/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa.gz' --gtf 'ftp://ftp.ensembl.org/pub/release-99/gtf/rattus_norvegicus/Rattus_norvegicus.Rnor_6.0.99.gtf.gz' --fc_count_type transcript {code} One major benefit of using this pipeline is that there is a lot of support for it in the larger bioinformatics community. There's a company (Sequera) that is supporting this and other Nextflow related projects, and |
Nextflow provides a standard type of RNA-Seq data pipeline called "nf-core/rnaseq" that we can potentially use for processing data in the Pollen PGRP and other related projects.
For example, the following command shows an example of running this pipeline on a "single end" data set, using a reference genome indicated by the "--fasta" option: {code} nextflow run nf-core/rnaseq -profile conda --singleEnd --reverseStranded --skipTrimming --reads '*.fastq.gz' --fasta 'ftp://ftp.ensembl.org/pub/release-99/fasta/rattus_norvegicus/dna_index/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa.gz' --gtf 'ftp://ftp.ensembl.org/pub/release-99/gtf/rattus_norvegicus/Rattus_norvegicus.Rnor_6.0.99.gtf.gz' --fc_count_type transcript {code} One major benefit of using this pipeline is that there is a lot of support for it in the larger bioinformatics community. There's a company (Sequera) that is supporting this and other Nextflow related projects. |
| Status | First Level Review in Progress [ 10301 ] | Ready for Pull Request [ 10304 ] |
| Status | Ready for Pull Request [ 10304 ] | Pull Request Submitted [ 10101 ] |
| Status | Pull Request Submitted [ 10101 ] | Reviewing Pull Request [ 10303 ] |
| Status | Reviewing Pull Request [ 10303 ] | Merged Needs Testing [ 10002 ] |
| Status | Merged Needs Testing [ 10002 ] | Post-merge Testing In Progress [ 10003 ] |
| Resolution | Done [ 10000 ] | |
| Status | Post-merge Testing In Progress [ 10003 ] | Closed [ 6 ] |
| Assignee | Nowlan Freese [ nfreese ] | Ann Loraine [ aloraine ] |
Added token and drive information to rclone configuration file on cluster.
Started copying data:
[aloraine@str-i1 nfcore_rnaseq]$ targ="brown:Experiments/ARE-WE-THERE-YET In vitro RNAseq #2 - time course 28-34/2021-09-28-nfcore-rnaseq-pipeline-run" [aloraine@str-i1 nfcore_rnaseq]$ echo $targ brown:Experiments/ARE-WE-THERE-YET In vitro RNAseq #2 - time course 28-34/2021-09-28-nfcore-rnaseq-pipeline-run [aloraine@str-i1 nfcore_rnaseq]$ rclone lsd "$targ" [ ... listed nothing because folder is empty ... ] [aloraine@str-i1 nfcore_rnaseq]$ rclone copy results "$targ"
| Link | This issue relates to IGBF-2969 [ IGBF-2969 ] |
| Link | This issue relates to IGBF-2971 [ IGBF-2971 ] |
See:
https://app.slack.com/client/TE6CZUZPH/CE8SSJV3N