Details
-
Type:
Task
-
Status: Closed (View Workflow)
-
Priority:
Trivial
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:None
-
Story Points:2
-
Epic Link:
-
Sprint:Spring 3, Spring 4, Spring 5, Spring 6
Description
Since we have consumed 83TB of storage space on the cluster, it is likely that many things can be removed to free up space.
We have done many runs on some datasets and we only really care about the runs that lead to IDB deposition and are on data that is POST SRA submission.
The goal of this ticket is to identify candidate folders that may be suitable for removal.
We will make a list of these folders. We will then have them reviewed by Ann, Molly and Rob to ensure nothing of importance get removed!
The list can be posted in the comments, with the exact cluster location.
Attachments
Activity
For now, we have plenty of space available.
13T.
/projects/tomato_genome 90T 78T 13T 86% aqu-fs19 171T 123T 49T 72%
We can revisit later when we consume this space.
For review:
Someone should read over these notes and ensure it makes sense. And then we close the ticket.
Chopping out an old analysis that I did.
sra44-apr2020
540K checkbams
48K lightHarvestingComplex
92M megamega
8.1T rnaseq-phase1
6.6T sra44-apr2020
4.0K sra44-sra22-pluscounts.txt
28M timecluster
We have since ran this one datasets many other ways, and it not needed. Free up 6.6
TB !!
And it is 4 years old.
Of the 80T, this is a breakdown of the folders:
8.3T alt_splicing
294G annswork
51T dataprocessing
2.5G db
3.3T genome
1.2T nfcore_rnaseq
1.5T rnaseq-phase2-areualive
15T robswork
9.9G scripts
0 ZZZ_pending_deletion
The top folders:
/projects/tomato_genome/rnaseq
3.4T
/projects/tomato_genome/SV100
0
/projects/tomato_genome/genome$ du -sh
71G
/projects/tomato_genome/db$ du -sh
718G
/projects/tomato_genome/sw$ du -sh
298M
/projects/tomato_genome/scripts$ du -sh
4.9G
/projects/tomato_genome/fnb
80T
So almost all of the the data is in fnb as expected.
Let's ignore the other folders and focus on just that.
A quick glance at the folders in
/projects/tomato_genome$
du -sh
84T
There are many files I don't have access to still. Likely these are Ann controlled folders.
Next, need to get size of each folder to see where most of our data is.
Used 'chmod -R g+w *' command on all of the 'data processing' directories that I have permissions too.
Some files I cannot see due to permissions.
.fnb/dataprocessing/*
I am guessing that these are Molly files and folders!
Can we get those permissions changed?
Outside of those folders,
we have 56,698 files in /projects/tomato_genome. !!!!
Everything makes sense and is a good place to end the ticket.