Uploaded image for project: 'IGB'
  1. IGB
  2. IGBF-1404

Restore GNC and Ecker ChIP-Seq data

    Details

    • Story Points:
      1
    • Sprint:
      Fall 2018 Sprint 2, Fall 2018 Sprint 3

      Description

      Identify GNC & Ecker data sets on HPC cluster and deploy to new ChIP-Seq data repository.

      Note: data may still be available on old transvar Andrew File System previously hosted in DHMRI server room. Data may also be available on transvar VM Blanchard set up following loss of DHMRI server room. Consult with Steven Blanchard to obtain backups, if they exist.

      When complete, be sure to update basecamp project(s) about where to find data.

        Attachments

          Issue Links

            Activity

            ann.loraine Ann Loraine created issue -
            ann.loraine Ann Loraine made changes -
            Field Original Value New Value
            Link This issue relates to IGBF-1403 [ IGBF-1403 ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            ann.loraine Ann Loraine made changes -
            Description Identify GCN & Ecker data sets on HPC cluster and deploy to new ChIP-Seq data repository.

            Note: data may still be available on old transvar systems previously hosted in DHMRI server room. Consult with Steven Blanchard to obtain backups, if they exist.
            Identify GCN & Ecker data sets on HPC cluster and deploy to new ChIP-Seq data repository.

            Note: data may still be available on old transvar Andrew File System previously hosted in DHMRI server room. Consult with Steven Blanchard to obtain backups, if they exist. Data may also be available on transvar VM Blanchard set up following loss of DHMRI server room.
            ann.loraine Ann Loraine made changes -
            Description Identify GCN & Ecker data sets on HPC cluster and deploy to new ChIP-Seq data repository.

            Note: data may still be available on old transvar Andrew File System previously hosted in DHMRI server room. Consult with Steven Blanchard to obtain backups, if they exist. Data may also be available on transvar VM Blanchard set up following loss of DHMRI server room.
            Identify GCN & Ecker data sets on HPC cluster and deploy to new ChIP-Seq data repository.

            Note: data may still be available on old transvar Andrew File System previously hosted in DHMRI server room. Data may also be available on transvar VM Blanchard set up following loss of DHMRI server room. Consult with Steven Blanchard to obtain backups, if they exist.
            ieclabau Ivory Blakley (Inactive) made changes -
            Summary Restore GCN and Ecker ChIP-Seq data Restore GNC and Ecker ChIP-Seq data
            ieclabau Ivory Blakley (Inactive) made changes -
            Description Identify GCN & Ecker data sets on HPC cluster and deploy to new ChIP-Seq data repository.

            Note: data may still be available on old transvar Andrew File System previously hosted in DHMRI server room. Data may also be available on transvar VM Blanchard set up following loss of DHMRI server room. Consult with Steven Blanchard to obtain backups, if they exist.
            Identify GNC & Ecker data sets on HPC cluster and deploy to new ChIP-Seq data repository.

            Note: data may still be available on old transvar Andrew File System previously hosted in DHMRI server room. Data may also be available on transvar VM Blanchard set up following loss of DHMRI server room. Consult with Steven Blanchard to obtain backups, if they exist.
            Hide
            ieclabau Ivory Blakley (Inactive) added a comment -

            I sent Steven an email.

            Show
            ieclabau Ivory Blakley (Inactive) added a comment - I sent Steven an email.
            ieclabau Ivory Blakley (Inactive) made changes -
            Sprint Fall 2018 Sprint 2 [ 52 ]
            ieclabau Ivory Blakley (Inactive) made changes -
            Rank Ranked lower
            ieclabau Ivory Blakley (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            ieclabau Ivory Blakley (Inactive) added a comment -

            Stephen replied. There is a backup.

            Show
            ieclabau Ivory Blakley (Inactive) added a comment - Stephen replied. There is a backup.
            Hide
            ieclabau Ivory Blakley (Inactive) added a comment -

            I copied the backup to my local machine using sftp.

            I created an EBS volume and attached it to my test quickload EC2 instance (http://18.222.191.240/).

            I copied everything but the ARR10_cytokinin project into this EBS volume.
            (this means I will need to update to the annots file to not include that project.)

            Show
            ieclabau Ivory Blakley (Inactive) added a comment - I copied the backup to my local machine using sftp. I created an EBS volume and attached it to my test quickload EC2 instance ( http://18.222.191.240/ ). I copied everything but the ARR10_cytokinin project into this EBS volume. (this means I will need to update to the annots file to not include that project.)
            Hide
            ieclabau Ivory Blakley (Inactive) added a comment -

            I moved the EBS volume within the EC2 instance to a web-accessible directory.
            The restored schallerlab quickload can now be added to IGB data sources using: http://18.222.191.240/Quickload_schallerlab/

            I need to add password protection.
            And I need to replace the current annots file with the reduced version (no ARR10_cytokinin).
            Then this will be in a state that Eric could use in the short term.

            I will need to separate out the GNC files with an annots file for just that project for Ann to add to the chip-seq repository.

            Show
            ieclabau Ivory Blakley (Inactive) added a comment - I moved the EBS volume within the EC2 instance to a web-accessible directory. The restored schallerlab quickload can now be added to IGB data sources using: http://18.222.191.240/Quickload_schallerlab/ I need to add password protection. And I need to replace the current annots file with the reduced version (no ARR10_cytokinin). Then this will be in a state that Eric could use in the short term. I will need to separate out the GNC files with an annots file for just that project for Ann to add to the chip-seq repository.
            ieclabau Ivory Blakley (Inactive) made changes -
            Description Identify GNC & Ecker data sets on HPC cluster and deploy to new ChIP-Seq data repository.

            Note: data may still be available on old transvar Andrew File System previously hosted in DHMRI server room. Data may also be available on transvar VM Blanchard set up following loss of DHMRI server room. Consult with Steven Blanchard to obtain backups, if they exist.
            Identify GNC & Ecker data sets on HPC cluster and deploy to new ChIP-Seq data repository.

            Note: data may still be available on old transvar Andrew File System previously hosted in DHMRI server room. Data may also be available on transvar VM Blanchard set up following loss of DHMRI server room. Consult with Steven Blanchard to obtain backups, if they exist.

            When complete, be sure to update basecamp project(s) about where to find data.
            ann.loraine Ann Loraine made changes -
            Assignee Ivory Blakley [ ieclabau ]
            ann.loraine Ann Loraine made changes -
            Assignee Ivory Blakley [ ieclabau ]
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            ann.loraine Ann Loraine made changes -
            Sprint Fall 2018 Sprint 2 [ 52 ] Fall 2018 Sprint 2, Fall 2018 Sprint 3 [ 52, 53 ]
            Hide
            ieclabau Ivory Blakley (Inactive) added a comment -

            At this point, I have exchanged several emails with Ann and Eric.
            The ARR10 ethylene dataset is still in some flux. It will probably move to RECI without password protection, but Eric still needs to approve that plan (awaiting reply email).
            The GNC and Ecker ChIP-Seq data are on the EBS volume, for testing and they look good. They are available on the HPC cluster under
            /nobackup/lorainelab/data/GNCChIP-Seq/Quickload_Site/Quickload_GNC
            and
            /nobackup/lorainelab/data/schallerlab_quickloads/Quickload_Ein3
            respectively.
            From there Ann can move them to the new chip seq quickload repository.

            Show
            ieclabau Ivory Blakley (Inactive) added a comment - At this point, I have exchanged several emails with Ann and Eric. The ARR10 ethylene dataset is still in some flux. It will probably move to RECI without password protection, but Eric still needs to approve that plan (awaiting reply email). The GNC and Ecker ChIP-Seq data are on the EBS volume, for testing and they look good. They are available on the HPC cluster under /nobackup/lorainelab/data/GNCChIP-Seq/Quickload_Site/Quickload_GNC and /nobackup/lorainelab/data/schallerlab_quickloads/Quickload_Ein3 respectively. From there Ann can move them to the new chip seq quickload repository.
            Hide
            ieclabau Ivory Blakley (Inactive) added a comment -

            I'm moving this to "needs first level review" and keeping it assigned to myself.
            That seems fitting because I'm doing working on it, but I still need to hear back from Eric and Ann about the final details before I can call it "done", and they could conceivably kick it back for changes.

            Show
            ieclabau Ivory Blakley (Inactive) added a comment - I'm moving this to "needs first level review" and keeping it assigned to myself. That seems fitting because I'm doing working on it, but I still need to hear back from Eric and Ann about the final details before I can call it "done", and they could conceivably kick it back for changes.
            ieclabau Ivory Blakley (Inactive) made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            GNC, EIN3 data added to: http://lorainelab-quickload.scidas.org/chipseq/
            Moved unpublished data to new hosting (ask Dr. Loraine for URL)

            Show
            ann.loraine Ann Loraine added a comment - - edited GNC, EIN3 data added to: http://lorainelab-quickload.scidas.org/chipseq/ Moved unpublished data to new hosting (ask Dr. Loraine for URL)
            ann.loraine Ann Loraine made changes -
            Status Needs 1st Level Review [ 10005 ] Needs Testing [ 10002 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ivory Blakley [ ieclabau ]
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            To test:

            • Add above link as new data source using IGB Data Access tab
            • Visit Arabidopsis genome
            • Open ChIP-Seq data set
            • Check that all files in the following data set folders can be loaded.

            Data sets to check:

            1. GNC PRJNA435725
            2. EIN3 PRJNA185404
            3. ARR10 PRJNA263839

            If any files cannot be loaded, report them here. In your report, include:

            • The path to the physical file on the host (file attribute in annots.xml)
            • The data set name as listed in the IGB user interface (title attribute in annots.xml)

            The "annots.xml" file is here: http://lorainelab-quickload.scidas.org/chipseq/A_thaliana_Jun_2009/annots.xml

            Show
            ann.loraine Ann Loraine added a comment - - edited To test: Add above link as new data source using IGB Data Access tab Visit Arabidopsis genome Open ChIP-Seq data set Check that all files in the following data set folders can be loaded. Data sets to check: GNC PRJNA435725 EIN3 PRJNA185404 ARR10 PRJNA263839 If any files cannot be loaded, report them here. In your report, include: The path to the physical file on the host (file attribute in annots.xml) The data set name as listed in the IGB user interface (title attribute in annots.xml) The "annots.xml" file is here: http://lorainelab-quickload.scidas.org/chipseq/A_thaliana_Jun_2009/annots.xml
            ann.loraine Ann Loraine made changes -
            Rank Ranked higher
            ann.loraine Ann Loraine made changes -
            Assignee Sneha Ramesh Watharkar [ jdaly ]
            ann.loraine Ann Loraine made changes -
            Assignee Sneha Ramesh Watharkar [ jdaly ]
            ptambvek Pranav Sanjay Tambvekar (Inactive) made changes -
            Status Needs Testing [ 10002 ] Testing In Progress [ 10003 ]
            ptambvek Pranav Sanjay Tambvekar (Inactive) made changes -
            Assignee Pranav Sanjay Tambvekar [ ptambvek ]
            Hide
            ptambvek Pranav Sanjay Tambvekar (Inactive) added a comment -

            While testing, error message as given below was shown on IGB:

            Could not load feature:
            The feature http://lorainelab-quickload.scidas.org/chipseq/A_thaliana_Jun_2009/ARR10_SRP048935/SRR1613114.bam is not reachable.

            1. file name= "ARR10_SRP048935/SRR1612835.bam"
            title="ARR10 PRJNA263839/alignments/line a1 BA-treated ChIP rep1 alignments"

            2. file name="ARR10_SRP048935/SRR1613114.bam"
            title="ARR10 PRJNA263839/alignments/line a1 BA-treated non-IP rep1 alignments"

            3. file name="ARR10_SRP048935/SRR1613116.bam"
            title="ARR10 PRJNA263839/alignments/line a1 BA-treated ChIP rep2 alignments"

            4. file name="ARR10_SRP048935/SRR1613117.bam"
            title="ARR10 PRJNA263839/alignments/line a1 BA-treated non-IP rep2 alignments"

            Show
            ptambvek Pranav Sanjay Tambvekar (Inactive) added a comment - While testing, error message as given below was shown on IGB: Could not load feature: The feature http://lorainelab-quickload.scidas.org/chipseq/A_thaliana_Jun_2009/ARR10_SRP048935/SRR1613114.bam is not reachable. 1. file name= "ARR10_SRP048935/SRR1612835.bam" title="ARR10 PRJNA263839/alignments/line a1 BA-treated ChIP rep1 alignments" 2. file name="ARR10_SRP048935/SRR1613114.bam" title="ARR10 PRJNA263839/alignments/line a1 BA-treated non-IP rep1 alignments" 3. file name="ARR10_SRP048935/SRR1613116.bam" title="ARR10 PRJNA263839/alignments/line a1 BA-treated ChIP rep2 alignments" 4. file name="ARR10_SRP048935/SRR1613117.bam" title="ARR10 PRJNA263839/alignments/line a1 BA-treated non-IP rep2 alignments"
            ptambvek Pranav Sanjay Tambvekar (Inactive) made changes -
            Status Testing In Progress [ 10003 ] Open [ 1 ]
            ptambvek Pranav Sanjay Tambvekar (Inactive) made changes -
            Assignee Pranav Sanjay Tambvekar [ ptambvek ] Ann Loraine [ aloraine ]
            Hide
            ann.loraine Ann Loraine added a comment -

            Thanks for catching this Pranav!
            For future reference, problem was due to incorrect file permissions setting - files above were not world-readable.
            This is now fixed.

            Show
            ann.loraine Ann Loraine added a comment - Thanks for catching this Pranav! For future reference, problem was due to incorrect file permissions setting - files above were not world-readable. This is now fixed.
            ann.loraine Ann Loraine made changes -
            Status Open [ 1 ] Reviewing Pull Request [ 10303 ]
            ann.loraine Ann Loraine made changes -
            Resolution Done [ 10000 ]
            Status Reviewing Pull Request [ 10303 ] Closed [ 6 ]
            ann.loraine Ann Loraine made changes -
            Workflow Loraine Lab Workflow [ 18109 ] Fall 2019 Workflow Update [ 19944 ]
            ann.loraine Ann Loraine made changes -
            Workflow Fall 2019 Workflow Update [ 19944 ] Revised Fall 2019 Workflow Update [ 22064 ]

              People

              • Assignee:
                ann.loraine Ann Loraine
                Reporter:
                ann.loraine Ann Loraine
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: