Details
-
Type:
Bug
-
Status: Closed (View Workflow)
-
Priority:
Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:None
-
Story Points:1
-
Sprint:Fall 7 : 11 Nov to 22 Nov, Fall 8 : 25 Nov to 6 Dec, Fall 9 : 9 Dec to 20 Dec, Spring 1 : 25 Dec to 17 Jan, Spring 2 : 20 Jan to 31 Jan, Spring 3 : 3 Feb to 14 Feb
Description
Situation: The two current analyses are working, however, I'm seeing a number of errors in the logs. After running several analyses successfully, the server stopped sending new analyses.
Task: Investigate the errors.
Attachments
Activity
Chaitanya Kintali, good to know, I will take a look and figure out what is going on with the folder. Thank you for looking into this.
Nowlan Freese, I have been testing various files in "/iplant/home/nowlanf/testFiles/" location. I found out, none of the files from that folder location are able to run successfully for any analysis. To confirm this behaviour i have copied the Bam_HomoSapien.bam(which was working fine) from SmokeTesting folder to testFiles folder. the analysis failed. and I have tweaked the code to output all the output files to nowlanf/output folder and observed all the analysis were able to run successfully. Clearly, the issue here is folder, especially when an app trying to output files to testFiles folder locaiton.
[Thu Nov 07 15:59:44.805426 2019] [wsgi:error] [pid 25693:tid 140656807245568] [remote 152.15.112.252:8686] {'AppId': 'c53d8c38-9c01-11e9-b4b7-008cfa5ae621', 'outputdir': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/', 'config': {'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c544a220-9c01-11e9-b4b7-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bam', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54dfb54-9c01-11e9-b4b7-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bw', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54f1160-9c01-11e9-b4b7-008cfa5ae621': 'bigwig', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe353de-9dbd-11e9-b934-008cfa5ae621': 'Counts Per Million', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe6070a-9dbd-11e9-b934-008cfa5ae621': '1', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_f418f434-9c10-11e9-89cd-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bai'}}
[Thu Nov 07 16:23:25.278615 2019] [wsgi:error] [pid 25693:tid 140656714909440] [remote 152.15.112.252:2472] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 16:25:43.018471 2019] [wsgi:error] [pid 25693:tid 140656714909440] [remote 152.15.112.252:32446] {'AppId': 'c53d8c38-9c01-11e9-b4b7-008cfa5ae621', 'outputdir': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/', 'config': {'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c544a220-9c01-11e9-b4b7-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bam', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54dfb54-9c01-11e9-b4b7-008cfa5ae621': 'test', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54f1160-9c01-11e9-b4b7-008cfa5ae621': 'bedgraph', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe353de-9dbd-11e9-b934-008cfa5ae621': 'Counts Per Million', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe6070a-9dbd-11e9-b934-008cfa5ae621': '100', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_f418f434-9c10-11e9-89cd-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bai'}}
[Thu Nov 07 16:49:03.088007 2019] [wsgi:error] [pid 25693:tid 140656714909440] [remote 152.15.112.252:11812] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 18:15:16.861892 2019] [wsgi:error] [pid 25693:tid 140656714909440] [remote 152.15.112.252:2880] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 18:28:51.950069 2019] [wsgi:error] [pid 25693:tid 140656706508544] [remote 152.15.112.252:47878] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 18:44:08.296627 2019] [wsgi:error] [pid 25693:tid 140656706508544] [remote 152.15.112.252:39501] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 18:45:18.709275 2019] [wsgi:error] [pid 25693:tid 140656714909440] [remote 152.15.112.252:19499] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 18:46:53.212253 2019] [wsgi:error] [pid 25693:tid 140656706508544] [remote 152.15.112.252:25533] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 18:48:29.069175 2019] [wsgi:error] [pid 25693:tid 140656706508544] [remote 152.15.112.252:16756] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 19:03:29.544546 2019] [wsgi:error] [pid 25693:tid 140656706508544] [remote 152.15.112.252:30393] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 19:03:36.565164 2019] [wsgi:error] [pid 25693:tid 140656714909440] [remote 152.15.112.252:38321] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 19:03:43.711208 2019] [wsgi:error] [pid 25693:tid 140656706508544] [remote 152.15.112.252:50904] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 19:04:21.382409 2019] [wsgi:error] [pid 25693:tid 140656714909440] [remote 152.15.112.252:28916] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 19:04:57.600089 2019] [wsgi:error] [pid 26933:tid 140062511605504] [remote 152.15.112.252:37727] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 19:05:12.777239 2019] [wsgi:error] [pid 26933:tid 140062436071168] [remote 152.15.112.252:56744] [<AppDetails: AppDetails object (91c41516-8c89-11e9-890e-008cfa5ae621)>, <AppDetails: AppDetails object (c53d8c38-9c01-11e9-b4b7-008cfa5ae621)>]
[Thu Nov 07 19:05:22.972915 2019] [wsgi:error] [pid 26933:tid 140062436071168] [remote 152.15.112.252:11352] {'AppId': '91c41516-8c89-11e9-890e-008cfa5ae621', 'outputdir': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/', 'config': {'91c4fecc-8c89-11e9-890e-008cfa5ae621_91c694bc-8c89-11e9-890e-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bam', '91c4fecc-8c89-11e9-890e-008cfa5ae621_bd9d0bba-8d1b-11e9-8836-008cfa5ae621': 'test6'}}
[Thu Nov 07 19:06:01.545578 2019] [wsgi:error] [pid 26933:tid 140062436071168] [remote 152.15.112.252:9321] {'AppId': 'c53d8c38-9c01-11e9-b4b7-008cfa5ae621', 'outputdir': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/', 'config': {'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c544a220-9c01-11e9-b4b7-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bam', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54dfb54-9c01-11e9-b4b7-008cfa5ae621': 'test8', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54f1160-9c01-11e9-b4b7-008cfa5ae621': 'bedgraph', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe353de-9dbd-11e9-b934-008cfa5ae621': 'None', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe6070a-9dbd-11e9-b934-008cfa5ae621': '100', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_f418f434-9c10-11e9-89cd-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bai'}}
[Thu Nov 07 19:06:26.248756 2019] [wsgi:error] [pid 26933:tid 140062444463872] [remote 152.15.112.252:21287] {'AppId': 'c53d8c38-9c01-11e9-b4b7-008cfa5ae621', 'outputdir': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/', 'config': {'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c544a220-9c01-11e9-b4b7-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bam', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54dfb54-9c01-11e9-b4b7-008cfa5ae621': 'test9-bg-cpm-100.bedgraph', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54f1160-9c01-11e9-b4b7-008cfa5ae621': 'bedgraph', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe353de-9dbd-11e9-b934-008cfa5ae621': 'Counts Per Million', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe6070a-9dbd-11e9-b934-008cfa5ae621': '100', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_f418f434-9c10-11e9-89cd-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bai'}}
[Thu Nov 07 19:06:43.213126 2019] [wsgi:error] [pid 26933:tid 140062436071168] [remote 152.15.112.252:49843] {'AppId': 'c53d8c38-9c01-11e9-b4b7-008cfa5ae621', 'outputdir': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/', 'config': {'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c544a220-9c01-11e9-b4b7-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bam', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54dfb54-9c01-11e9-b4b7-008cfa5ae621': 'test10-bw-cpm-100.bw', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54f1160-9c01-11e9-b4b7-008cfa5ae621': 'bigwig', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe353de-9dbd-11e9-b934-008cfa5ae621': 'Counts Per Million', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe6070a-9dbd-11e9-b934-008cfa5ae621': '100', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_f418f434-9c10-11e9-89cd-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bai'}}
[Thu Nov 07 19:06:59.441630 2019] [wsgi:error] [pid 26933:tid 140062444463872] [remote 152.15.112.252:13048] {'AppId': 'c53d8c38-9c01-11e9-b4b7-008cfa5ae621', 'outputdir': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/', 'config': {'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c544a220-9c01-11e9-b4b7-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bam', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54dfb54-9c01-11e9-b4b7-008cfa5ae621': 'test10-bg-none-100.bedgraph', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_c54f1160-9c01-11e9-b4b7-008cfa5ae621': 'bigwig', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe353de-9dbd-11e9-b934-008cfa5ae621': 'None', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_cbe6070a-9dbd-11e9-b934-008cfa5ae621': '100', 'c53e2896-9c01-11e9-b4b7-008cfa5ae621_f418f434-9c10-11e9-89cd-008cfa5ae621': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/Bam_HomoSapien.bai'}}
[Thu Nov 07 19:07:16.957580 2019] [wsgi:error] [pid 26933:tid 140062436071168] [remote 152.15.112.252:45804] {'AppId': 'c53d8c38-9c01-11e9-b4b7-008cfa5ae621', 'outputdir': '/iplant/home/nowlanf/SmokeTesting/H_sapiens_Dec_2013/', 'config':
}
This issue related to logging statements in the code used for testing analysis submissions - these were not necessarily actual errors being logged. The other issue of analyses failing in specific directories will be addressed in
IGBF-2119which contains multiple analysis issues to be addressed.