I freed up about 26TB. We are using 64TB with /projects/tomato_genome/. The work directories after a next flow run can eat up space.

URC updated the folders, now we have full access!

An easy way to free up space is to remove the work folders in each nextflow run.
This can free up to 1.2 TB per nextflow run!!!

Going to assign to Brandon.

I got word from URC that all permissions have been changed. However, when I checked online, there were still a handful of folders that I could not access.
I pinged URC with a specific example and inquired as to what was happening.

R

The largest folders in this tomato_genome space:

93067638872 /projects/tomato_genome
88620314732 /projects/tomato_genome/fnb
67754157536 /projects/tomato_genome/fnb/dataprocessing
9736744168 /projects/tomato_genome/fnb/dataprocessing/SRP460750
9143198808 /projects/tomato_genome/fnb/dataprocessing/muday-144
8052284232 /projects/tomato_genome/fnb/alt_splicing
6273126216 /projects/tomato_genome/fnb/robswork
6273003708 /projects/tomato_genome/fnb/robswork/rnaseq-phase1
5911039864 /projects/tomato_genome/fnb/dataprocessing/trinity
5525239128 /projects/tomato_genome/fnb/dataprocessing/brandon_work
5022973880 /projects/tomato_genome/fnb/dataprocessing/seedlingPollen
4971115136 /projects/tomato_genome/fnb/dataprocessing/30-804059537
4970046188 /projects/tomato_genome/fnb/dataprocessing/muday-144/nfcore-SL5
4845797268 /projects/tomato_genome/fnb/dataprocessing/seedlingPollen/nfcore-SL5
4273838832 /projects/tomato_genome/fnb/dataprocessing/SRP486761
3942498252 /projects/tomato_genome/fnb/dataprocessing/SRP438952
3607916196 /projects/tomato_genome/rnaseq
3586508556 /projects/tomato_genome/fnb/alt_splicing/SRP268884
3530400688 /projects/tomato_genome/fnb/dataprocessing/30-804059537/S_lycopersicum_Jun_2022
3458070616 /projects/tomato_genome/fnb/genome
3382991628 /projects/tomato_genome/fnb/dataprocessing/SRP487154
3373876652 /projects/tomato_genome/fnb/dataprocessing/brandon_work/NEXTFLOW
3356222392 /projects/tomato_genome/fnb/dataprocessing/30-681594536
3094954620 /projects/tomato_genome/fnb/dataprocessing/muday-144/nfcore-SL5/kanhunt
2675755408 /projects/tomato_genome/fnb/dataprocessing/SRP460750/nfcore-SL5-2024-05-07
2623094696 /projects/tomato_genome/fnb/dataprocessing/SRP460750/nfcore-SL4-2024-05-21
2478807264 /projects/tomato_genome/fnb/dataprocessing/SRP438952/nfcore-SL4
2405007848 /projects/tomato_genome/fnb/dataprocessing/seedlingPollen/nfcore-SL5/work
2154415108 /projects/tomato_genome/fnb/dataprocessing/SRP486761/nfcore-SL5
2146180328 /projects/tomato_genome/fnb/dataprocessing/tmp-splice
2119423724 /projects/tomato_genome/fnb/dataprocessing/SRP486761/nfcore-SL4
2115371444 /projects/tomato_genome/fnb/dataprocessing/SRP441343
2072056112 /projects/tomato_genome/fnb/dataprocessing/SRP438952/nfcore-SL4/work
2050700260 /projects/tomato_genome/fnb/robswork/rnaseq-phase1/mitochloro
2026976592 /projects/tomato_genome/fnb/dataprocessing/SRP460750/nfcore-SL5

Made a request to the URC to get Molly permissions changed to be group writable and readable.

Side note: Trinity and most things from the Broad Institute create MASSIVE amounts of small temp files.

In this spaec,
/projects/tomato_genome/fnb/robswork/rnaseq-phase1$ rm -R trinity/

I have started to remove an old Trinity run. It has been running for 20 minutes to erase these temp folders and files.
going forward when running Trinity for de novo assembly, we should add a command to remove the temp files soon after the runs are complete.

First issue is that we have MANY files generated by Molly that have the incorrect permissions as part of NFCORE.

So we need group access to MANY files! This might be a job for the URC folks.

I will touch base with Steven first and see if there are better strategies to employ.

To identify biggest files:
du -a /projects/tomato_genome | sort -n -r | head -n 100 > largestFiles.txt