Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      Task: Add the Vanessa cardui genome and annotation to IGB.

      Link to genome on NCBI: https://www.ncbi.nlm.nih.gov/datasets/genome/GCF_905220365.1/

        Attachments

          Issue Links

            Activity

            pkulzer Paige Kulzer created issue -
            pkulzer Paige Kulzer made changes -
            Field Original Value New Value
            Epic Link IGBF-3823 [ 23122 ]
            pkulzer Paige Kulzer made changes -
            Description Task: Add the Vanessa cardui genome and annotation to IGB. Task: Add the Vanessa cardui genome and annotation to IGB.

            Link to genome on NCBI: https://www.ncbi.nlm.nih.gov/datasets/genome/GCF_905220365.1/
            Sprint Fall 2 [ 203 ]
            pkulzer Paige Kulzer made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            pkulzer Paige Kulzer added a comment -

            Below is an outline of the steps I followed to create this Quickload:

            1.Use wget to obtain the .2bit file from UCSC's track hub directory, then rename it

            wget https://hgdownload.soe.ucsc.edu/hubs/GCF/905/220/365/GCF_905220365.1/GCF_905220365.1.2bit
            mv GCF_905220365.1.2bit ilVanCard2.1.2bit
            

            2. Create genome.txt, then check that the chromosome's are ordered logically (i.e., numerically)

            ./twoBitInfo ilVanCard2.1.2bit genome.txt
            cat genome.txt
            

            3. Use Vanessa cardui's taxID (171605) to get the information needed from gene2accession.gz and gene_info.gz to create the BED14 file in a later step

            gunzip -c gene2accession.gz | grep '^171605\t' > 171605.gene2accession.txt
            gunzip -c gene_info.gz | grep '^171605\t' > 171605.gene_info.txt
            

            4. Download the RefSeqAll BED file from UCSC's table browser (Link: https://genome.ucsc.edu/cgi-bin/hgTables), then create the BED14 file using the following code:

            cd ~/Documents/Repos/genomesource/
            ./ucscToBedDetail.py -a ~/Downloads/171605.gene2accession.txt -g ~/Downloads/171605.gene_info.txt ~/Downloads/V_cardui_ncbiRefSeq.bed.gz ~/Downloads/V_cardui_Feb_2021_ncbiRefSeq.bed
            

            5. Sort, gzip, and tabix the BED14 file

            cd ~/Downloads/
            sort -k1,1 -k2,2n V_cardui_Feb_2021_ncbiRefSeq.bed | bgzip > V_cardui_Feb_2021_ncbiRefSeq.bed.gz
            tabix -0 -s 1 -b 2 -e 3 V_cardui_Feb_2021_ncbiRefSeq.bed.gz
            

            6. Sanity check the 2bit and BED files - Add the 2bit file as a reference, then drag/drop the BED files into IGB. Confirm that gene models are present, labeled correctly, and that no error messages are present in the Log.

            7. Create a new directory in the quickload repo, then create annots.xml

            cd ~/Documents/Repos/quickload/
            svn mkdir V_cardui_Feb_2021
            svn cp A_gambiae_Feb_2003/annots.xml V_cardui_Feb_2021
            nano V_cardui_Feb_2021/annots.xml
            

            8. Add V_cardui_Feb_2021 to contents.txt and .htaccess

            nano contents.txt
            

            V_cardui_Feb_2021 Vanessa cardui (Feb 2021) painted lady (ilVanCard2.1)

            nano .htaccess
            

            AddDescription "Vanessa cardui (Feb 2021) painted lady (ilVanCard2.1)" V_cardui_Feb_2021

            9. Create HEADER.md

            ../genomesource/writeQuickLoadHeaderUCSC.py V_cardui_Feb_2021 > V_cardui_Feb_2021/HEADER.md
            
            Show
            pkulzer Paige Kulzer added a comment - Below is an outline of the steps I followed to create this Quickload: 1.Use wget to obtain the .2bit file from UCSC's track hub directory, then rename it wget https: //hgdownload.soe.ucsc.edu/hubs/GCF/905/220/365/GCF_905220365.1/GCF_905220365.1.2bit mv GCF_905220365.1.2bit ilVanCard2.1.2bit 2. Create genome.txt, then check that the chromosome's are ordered logically (i.e., numerically) ./twoBitInfo ilVanCard2.1.2bit genome.txt cat genome.txt 3. Use Vanessa cardui 's taxID (171605) to get the information needed from gene2accession.gz and gene_info.gz to create the BED14 file in a later step gunzip -c gene2accession.gz | grep '^171605\t' > 171605.gene2accession.txt gunzip -c gene_info.gz | grep '^171605\t' > 171605.gene_info.txt 4. Download the RefSeqAll BED file from UCSC's table browser (Link: https://genome.ucsc.edu/cgi-bin/hgTables ), then create the BED14 file using the following code: cd ~/Documents/Repos/genomesource/ ./ucscToBedDetail.py -a ~/Downloads/171605.gene2accession.txt -g ~/Downloads/171605.gene_info.txt ~/Downloads/V_cardui_ncbiRefSeq.bed.gz ~/Downloads/V_cardui_Feb_2021_ncbiRefSeq.bed 5. Sort, gzip, and tabix the BED14 file cd ~/Downloads/ sort -k1,1 -k2,2n V_cardui_Feb_2021_ncbiRefSeq.bed | bgzip > V_cardui_Feb_2021_ncbiRefSeq.bed.gz tabix -0 -s 1 -b 2 -e 3 V_cardui_Feb_2021_ncbiRefSeq.bed.gz 6. Sanity check the 2bit and BED files - Add the 2bit file as a reference, then drag/drop the BED files into IGB. Confirm that gene models are present, labeled correctly, and that no error messages are present in the Log. 7. Create a new directory in the quickload repo, then create annots.xml cd ~/Documents/Repos/quickload/ svn mkdir V_cardui_Feb_2021 svn cp A_gambiae_Feb_2003/annots.xml V_cardui_Feb_2021 nano V_cardui_Feb_2021/annots.xml 8. Add V_cardui_Feb_2021 to contents.txt and .htaccess nano contents.txt V_cardui_Feb_2021 Vanessa cardui (Feb 2021) painted lady (ilVanCard2.1) nano .htaccess AddDescription "Vanessa cardui (Feb 2021) painted lady (ilVanCard2.1)" V_cardui_Feb_2021 9. Create HEADER.md ../genomesource/writeQuickLoadHeaderUCSC.py V_cardui_Feb_2021 > V_cardui_Feb_2021/HEADER.md
            Hide
            pkulzer Paige Kulzer added a comment - - edited

            For review: I have zipped up a folder containing all of the relevant files needed to create this Quickload and place that in the shared Google Drive. Please take a look at these files and load the 2bit and BED files in IGB to double check that they're working properly.

            Location of the Quickload on Google Drive:
            Path: research-big-lorainelab > IGB Project Documentation and Plans > IGB Genomes > V_cardui.zip
            Link: https://drive.google.com/drive/folders/1bFRx4PqldxNf400n7Vr9SD_dNeNmtpvk?usp=drive_link

            Show
            pkulzer Paige Kulzer added a comment - - edited For review: I have zipped up a folder containing all of the relevant files needed to create this Quickload and place that in the shared Google Drive. Please take a look at these files and load the 2bit and BED files in IGB to double check that they're working properly. Location of the Quickload on Google Drive: Path : research-big-lorainelab > IGB Project Documentation and Plans > IGB Genomes > V_cardui.zip Link : https://drive.google.com/drive/folders/1bFRx4PqldxNf400n7Vr9SD_dNeNmtpvk?usp=drive_link
            pkulzer Paige Kulzer made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            pkulzer Paige Kulzer made changes -
            Assignee Paige Kulzer [ pkulzer ] Nowlan Freese [ nfreese ]
            pkulzer Paige Kulzer made changes -
            Story Points 2 1
            pkulzer Paige Kulzer made changes -
            Rank Ranked higher
            nfreese Nowlan Freese made changes -
            Sprint Fall 2 [ 203 ] Fall 3 [ 204 ]
            nfreese Nowlan Freese made changes -
            Sprint Fall 3 [ 204 ] Fall 4 [ 205 ]
            nfreese Nowlan Freese made changes -
            Sprint Fall 4 [ 205 ] Fall 5 [ 206 ]
            nfreese Nowlan Freese made changes -
            Sprint Fall 5 [ 206 ] Fall 6 [ 207 ]
            nfreese Nowlan Freese made changes -
            Sprint Fall 6 [ 207 ] Fall 7 [ 208 ]
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ] Paige Kulzer [ pkulzer ]
            nfreese Nowlan Freese made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            nfreese Nowlan Freese made changes -
            Status First Level Review in Progress [ 10301 ] To-Do [ 10305 ]
            pkulzer Paige Kulzer made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            Hide
            pkulzer Paige Kulzer added a comment -

            I've edited the header to make it specific to NCBI rather than UCSC, added species.txt and synonyms.txt to the zip file, cut down contents.txt to only include this genome, and edited annots.txt to replace UCSC's "refGene" with NCBI's "refSeq".

            Let me know if I've updated all of these files correctly!

            Show
            pkulzer Paige Kulzer added a comment - I've edited the header to make it specific to NCBI rather than UCSC, added species.txt and synonyms.txt to the zip file, cut down contents.txt to only include this genome, and edited annots.txt to replace UCSC's "refGene" with NCBI's "refSeq". Let me know if I've updated all of these files correctly!
            pkulzer Paige Kulzer made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            pkulzer Paige Kulzer made changes -
            Assignee Paige Kulzer [ pkulzer ] Nowlan Freese [ nfreese ]
            nfreese Nowlan Freese made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            Hide
            nfreese Nowlan Freese added a comment -

            Testing: Everything looks good

            Show
            nfreese Nowlan Freese added a comment - Testing: Everything looks good
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ] Paige Kulzer [ pkulzer ]
            nfreese Nowlan Freese made changes -
            Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
            Hide
            pkulzer Paige Kulzer added a comment -

            The Vanessa cardui genome has been pushed to the SVN repo.

            Ready for final review!

            Show
            pkulzer Paige Kulzer added a comment - The Vanessa cardui genome has been pushed to the SVN repo. Ready for final review!
            pkulzer Paige Kulzer made changes -
            Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
            pkulzer Paige Kulzer made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            pkulzer Paige Kulzer made changes -
            Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
            pkulzer Paige Kulzer made changes -
            Assignee Paige Kulzer [ pkulzer ] Nowlan Freese [ nfreese ]
            Hide
            ann.loraine Ann Loraine added a comment -

            I have deployed the latest copy of quickload repository to:

            RENCI hosting - http://igbquickload-main.bioviz.org/quickload/ (primary)
            UNC Charlotte hosting - http://igbquickload.org/quickload/ (backup)
            To test:

            • launch IGB and visit each new genome version (see above)
            • visit the subdirectories for each genome (by following the links above) and check that there is text describing the genome and datasets visible in IGB itself
            • within IGB Available Data section, click any "linkout" icons and make sure a Web page opens and that it goes to a place that describes the dataset somehow
            • check that when the datasets load, they look OK - gene models should be boxes with lines connecting them, for instance, and the track labels should be readable and should make sense ("making sense" is a subjective of course! mainly we're looking for problems that could trip up a user and cause confusion.)
            Show
            ann.loraine Ann Loraine added a comment - I have deployed the latest copy of quickload repository to: RENCI hosting - http://igbquickload-main.bioviz.org/quickload/ (primary) UNC Charlotte hosting - http://igbquickload.org/quickload/ (backup) To test: launch IGB and visit each new genome version (see above) visit the subdirectories for each genome (by following the links above) and check that there is text describing the genome and datasets visible in IGB itself within IGB Available Data section, click any "linkout" icons and make sure a Web page opens and that it goes to a place that describes the dataset somehow check that when the datasets load, they look OK - gene models should be boxes with lines connecting them, for instance, and the track labels should be readable and should make sense ("making sense" is a subjective of course! mainly we're looking for problems that could trip up a user and cause confusion.)
            nfreese Nowlan Freese made changes -
            Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
            Hide
            nfreese Nowlan Freese added a comment -

            Paige Kulzer - the 2bit file has a weird name and isn't loading correctly, can you update it to V_cardui_Feb_2021.2bit and push it to the SVN repo then ask Dr. Loraine to deploy it?

            Show
            nfreese Nowlan Freese added a comment - Paige Kulzer - the 2bit file has a weird name and isn't loading correctly, can you update it to V_cardui_Feb_2021.2bit and push it to the SVN repo then ask Dr. Loraine to deploy it?
            nfreese Nowlan Freese made changes -
            Assignee Nowlan Freese [ nfreese ] Paige Kulzer [ pkulzer ]
            nfreese Nowlan Freese made changes -
            Status Post-merge Testing In Progress [ 10003 ] To-Do [ 10305 ]
            Hide
            pkulzer Paige Kulzer added a comment -

            Ann Loraine, as per Nowlan's comment above, I've updated the 2bit files name to "V_cardui_Feb_2021.2bit" and have pushed it to the SVN repo. It's ready for you to deploy!

            Show
            pkulzer Paige Kulzer added a comment - Ann Loraine , as per Nowlan's comment above, I've updated the 2bit files name to "V_cardui_Feb_2021.2bit" and have pushed it to the SVN repo. It's ready for you to deploy!
            pkulzer Paige Kulzer made changes -
            Status To-Do [ 10305 ] In Progress [ 3 ]
            pkulzer Paige Kulzer made changes -
            Status In Progress [ 3 ] Needs 1st Level Review [ 10005 ]
            pkulzer Paige Kulzer made changes -
            Status Needs 1st Level Review [ 10005 ] First Level Review in Progress [ 10301 ]
            pkulzer Paige Kulzer made changes -
            Status First Level Review in Progress [ 10301 ] Ready for Pull Request [ 10304 ]
            pkulzer Paige Kulzer made changes -
            Status Ready for Pull Request [ 10304 ] Pull Request Submitted [ 10101 ]
            pkulzer Paige Kulzer made changes -
            Assignee Paige Kulzer [ pkulzer ] Ann Loraine [ aloraine ]
            Hide
            ann.loraine Ann Loraine added a comment -

            Update:

            • New file is deployed and old one is deleted - I updated quickloads hosted at RENCI and UNC Charlotte
            Show
            ann.loraine Ann Loraine added a comment - Update: New file is deployed and old one is deleted - I updated quickloads hosted at RENCI and UNC Charlotte
            Hide
            ann.loraine Ann Loraine added a comment - - edited

            Testing:

            It looks like the documentation describing the data files needs to be improved.

            The description of the files and where they came from in the comments above don't match the documentation. Specifically, the origin of the files is different. It looks like we got the files from the UCSC Genome Informations Web site and then used our tools to convert them to our preferred formats.

            Suggestion:

            Since we are doing a pretty good job of describing our process in the comments above (thanks Paige Kulzer) could we simply link to this ticket from our header file?

            We could provide some very generic – but accurate – descriptions of where the files came from and how we processed them. We could also then provide a direct link to this Jira ticket, which should be publicly accessible.

            I recommend we maybe make "Improve documentation for painted lady genome assembly quickload" as a new ticket.

            Once that documentation is more complete, we could then get in touch with researchers working with this assembly and get their input. Also, we should make sure to link to articles for this particular genome assembly so that users will be able to get more information.

            Show
            ann.loraine Ann Loraine added a comment - - edited Testing: It looks like the documentation describing the data files needs to be improved. The description of the files and where they came from in the comments above don't match the documentation. Specifically, the origin of the files is different. It looks like we got the files from the UCSC Genome Informations Web site and then used our tools to convert them to our preferred formats. Suggestion: Since we are doing a pretty good job of describing our process in the comments above (thanks Paige Kulzer ) could we simply link to this ticket from our header file? We could provide some very generic – but accurate – descriptions of where the files came from and how we processed them. We could also then provide a direct link to this Jira ticket, which should be publicly accessible. I recommend we maybe make "Improve documentation for painted lady genome assembly quickload" as a new ticket. Once that documentation is more complete, we could then get in touch with researchers working with this assembly and get their input. Also, we should make sure to link to articles for this particular genome assembly so that users will be able to get more information.
            ann.loraine Ann Loraine made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            ann.loraine Ann Loraine made changes -
            Status Reviewing Pull Request [ 10303 ] To-Do [ 10305 ]
            ann.loraine Ann Loraine made changes -
            Assignee Ann Loraine [ aloraine ] Paige Kulzer [ pkulzer ]
            ann.loraine Ann Loraine made changes -
            Link This issue relates to IGBF-4038 [ IGBF-4038 ]
            Hide
            ann.loraine Ann Loraine added a comment -

            I made the recommend ticket for improving the genome assembly documentation. Closing this one now.

            Show
            ann.loraine Ann Loraine added a comment - I made the recommend ticket for improving the genome assembly documentation. Closing this one now.
            ann.loraine Ann Loraine made changes -
            Status To-Do [ 10305 ] Pull Request Submitted [ 10101 ]
            ann.loraine Ann Loraine made changes -
            Status Pull Request Submitted [ 10101 ] Reviewing Pull Request [ 10303 ]
            ann.loraine Ann Loraine made changes -
            Status Reviewing Pull Request [ 10303 ] Merged Needs Testing [ 10002 ]
            ann.loraine Ann Loraine made changes -
            Status Merged Needs Testing [ 10002 ] Post-merge Testing In Progress [ 10003 ]
            ann.loraine Ann Loraine made changes -
            Resolution Done [ 10000 ]
            Status Post-merge Testing In Progress [ 10003 ] Closed [ 6 ]

              People

              • Assignee:
                pkulzer Paige Kulzer
                Reporter:
                pkulzer Paige Kulzer
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: