INFO Sourcing add config line: /local/execute/dir_223879/glide_S5VAG8/add_config_line.source INFO Discarding path '/hadoop'. File does not exist INFO Discarding path '/hdfs'. File does not exist INFO Discarding path '/lizard'. File does not exist INFO Discarding path '/mnt/hadoop'. File does not exist INFO Discarding path '/mnt/hdfs'. File does not exist WARNING: Container does not have an exec helper script, calling '/srv/.gwms-user-job-wrapper.sh' directly WARNING glidein_config not definied () in singularity_lib.sh. Some functions like advertise and error_gen will be limited. =========================================================== There was a crash. This is the entire stack trace of all threads: =========================================================== Thread 8 (Thread 0x2b683ee95700 (LWP 252)): #0 0x00002b68279a7d12 in pthread_cond_timedwait GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000efbb1c in __gthread_cond_timedwait (__abs_timeout=0x2b683ee8e340, __mutex=0x2b684010a068, __cond=0x2b684010a090) at /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=, this=0x2b684010a090) at /usr/include/c++/4.8.2/condition_variable:160 #3 wait_until > > (__atime=..., __lock=, this=0x2b684010a090) at /usr/include/c++/4.8.2/condition_variable:100 #4 wait_for > (__rtime=..., __lock=, this=0x2b684010a090) at /usr/include/c++/4.8.2/condition_variable:132 #5 JEventSource_EVIOpp::GetEvent (this=0x2b6840109fa0, event=...) at libraries/DAQ/JEventSource_EVIOpp.cc:497 #6 0x00000000010a23f7 in jana::JEventSource::GetEvent (this=0x2b6840109fa0, event=...) at src/JANA/JEventSource.cc:54 #7 0x00000000010865ca in jana::JApplication::ReadEvent (this=0x7fff461b7420, event=...) at src/JANA/JApplication.cc:824 #8 0x0000000001086320 in jana::JApplication::EventBufferThread (this=this entry=0x7fff461b7420) at src/JANA/JApplication.cc:753 #9 0x00000000010865a9 in LaunchEventBufferThread (arg=0x7fff461b7420) at src/JANA/JApplication.cc:666 #10 0x00002b68279a3dd5 in start_thread () from /lib64/libpthread.so.0 #11 0x00002b68286d9ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x2b684b471700 (LWP 253)): #0 0x00002b68286a0e2d in nanosleep () from /lib64/libc.so.6 #1 0x00002b68286a0cc4 in sleep () from /lib64/libc.so.6 #2 0x00002b682af37f8c in GarbageCollectorThread (arg=0x2b6840006730, thr=) at /nfs/cern.ch/asis/xrootd-3.3.2/src/XrdClient/XrdClientConnMgr.cc:93 #3 0x00002b682b19208f in XrdSysThread_Xeq (myargs=0x2b68400089e0) at /nfs/cern.ch/asis/xrootd-3.3.2/src/XrdSys/XrdSysPthread.cc:86 #4 0x00002b68279a3dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00002b68286d9ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x2b684bc74700 (LWP 255)): #0 0x00002b68286a0e2d in nanosleep () from /lib64/libc.so.6 #1 0x00002b68286d1704 in usleep () from /lib64/libc.so.6 #2 0x000000000107810a in jana::JApplication::NextEvent (this=0x7fff461b7420, event=...) at src/JANA/JApplication.cc:614 #3 0x00000000010a0f1e in jana::JEventLoop::OneEvent (this=this entry=0x2b68500008c0) at src/JANA/JEventLoop.cc:591 #4 0x00000000010a1af4 in jana::JEventLoop::Loop (this=this entry=0x2b68500008c0) at src/JANA/JEventLoop.cc:496 #5 0x0000000001078daa in LaunchThread (arg=0x7fff461b7420) at src/JANA/JApplication.cc:1382 #6 0x00002b68279a3dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00002b68286d9ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x2b6854200700 (LWP 256)): #0 0x00002b68286cf20d in poll () from /lib64/libc.so.6 #1 0x00002b682af21591 in XrdClientSock::RecvRaw (this=0x2b684010fcb0, buffer=0x2b6858000900, length=8, substreamid=, usedsubstreamid=) at /nfs/cern.ch/asis/xrootd-3.3.2/src/XrdClient/XrdClientSock.cc:156 #2 0x00002b682af410af in XrdClientPhyConnection::ReadRaw (this=0x2b684010afc0, buf=0x2b6858000900, len=8, substreamid=-1, usedsubstreamid=0x2b68541f9068) at /nfs/cern.ch/asis/xrootd-3.3.2/src/XrdClient/XrdClientPhyConnection.cc:381 #3 0x00002b682af43b73 in XrdClientMessage::ReadRaw (this=0x2b68580008c0, phy=0x2b684010afc0) at /nfs/cern.ch/asis/xrootd-3.3.2/src/XrdClient/XrdClientMessage.cc:172 #4 0x00002b682af3fa58 in XrdClientPhyConnection::BuildMessage (this=0x2b684010afc0, IgnoreTimeouts=true, Enqueue=true) at /nfs/cern.ch/asis/xrootd-3.3.2/src/XrdClient/XrdClientPhyConnection.cc:462 #5 0x00002b682af4279a in SocketReaderThread (arg=0x2b684010afc0, thr=) at /nfs/cern.ch/asis/xrootd-3.3.2/src/XrdClient/XrdClientPhyConnection.cc:79 #6 0x00002b682b19208f in XrdSysThread_Xeq (myargs=0x2b684010ac90) at /nfs/cern.ch/asis/xrootd-3.3.2/src/XrdSys/XrdSysPthread.cc:86 #7 0x00002b68279a3dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00002b68286d9ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x2b6854401700 (LWP 260)): #0 0x00002b68286a0e2d in nanosleep () from /lib64/libc.so.6 #1 0x00002b68286d1704 in usleep () from /lib64/libc.so.6 #2 0x0000000000efc01d in sleep_for > (__rtime=...) at /usr/include/c++/4.8.2/thread:281 #3 JEventSource_EVIOpp::Dispatcher (this=0x2b6840109fa0) at libraries/DAQ/JEventSource_EVIOpp.cc:339 #4 0x00002b6827e71070 in ?? () from /lib64/libstdc++.so.6 #5 0x00002b68279a3dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00002b68286d9ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x2b6854602700 (LWP 261)): #0 0x00002b68286a0a89 in waitpid () from /lib64/libc.so.6 #1 0x00002b682861ede2 in do_system () from /lib64/libc.so.6 #2 0x00002b682861f191 in system () from /lib64/libc.so.6 #3 0x00002b6821f5f439 in TUnixSystem::StackTrace (this=0x324f230) at /group/halld/Software/builds/Linux_CentOS7-x86_64-gcc4.8.5-cntr/root/root-6.08.06/core/unix/src/TUnixSystem.cxx:2405 #4 0x00002b6821f61d0c in TUnixSystem::DispatchSignals (this=0x324f230, sig=kSigSegmentationViolation) at /group/halld/Software/builds/Linux_CentOS7-x86_64-gcc4.8.5-cntr/root/root-6.08.06/core/unix/src/TUnixSystem.cxx:3625 #5 #6 0x0000000000f3e9b8 in swap_block (outbuff=0x2b6940041000, len=2311061517, inbuff=0x2b6940041000) at libraries/DAQ/swap_bank.h:57 #7 swap_bank (outbuff=0x2b6870000010, inbuff=0x2b6870000010, len=) at libraries/DAQ/swap_bank.cc:51 #8 0x0000000000f1cfb8 in DEVIOWorkerThread::Run (this=0x2b68401136f0) at libraries/DAQ/DEVIOWorkerThread.cc:107 #9 0x00002b6827e71070 in ?? () from /lib64/libstdc++.so.6 #10 0x00002b68279a3dd5 in start_thread () from /lib64/libpthread.so.0 #11 0x00002b68286d9ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x2b6854803700 (LWP 266)): #0 0x00002b68279a7965 in pthread_cond_wait GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b6827e6d82c in std::condition_variable::wait(std::unique_lock&) () from /lib64/libstdc++.so.6 #2 0x0000000000f092ea in async_filebuf::readloop (this=0x2b6840004e30) at libraries/DAQ/async_filebuf.cc:104 #3 0x00002b6827e71070 in ?? () from /lib64/libstdc++.so.6 #4 0x00002b68279a3dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00002b68286d9ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x2b682c06cc00 (LWP 243)): #0 0x00002b68279aae3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000000000108214d in jana::JApplication::Run (this=this entry=0x7fff461b7420, proc=proc entry=0x0, Nthreads=, Nthreads entry=1) at src/JANA/JApplication.cc:1613 #2 0x000000000064162f in main (narg=17, argv=) at programs/Analysis/hd_ana/hd_ana.cc:24 =========================================================== The lines below might hint at the cause of the crash. You may get help by asking at the ROOT forum http://root.cern.ch/forum. Only if you are really convinced it is a bug in ROOT then please submit a report at http://root.cern.ch/bugs. Please post the ENTIRE stack trace from above as an attachment in addition to anything else that might help us fixing this issue. =========================================================== #6 0x0000000000f3e9b8 in swap_block (outbuff=0x2b6940041000, len=2311061517, inbuff=0x2b6940041000) at libraries/DAQ/swap_bank.h:57 #7 swap_bank (outbuff=0x2b6870000010, inbuff=0x2b6870000010, len=) at libraries/DAQ/swap_bank.cc:51 #8 0x0000000000f1cfb8 in DEVIOWorkerThread::Run (this=0x2b68401136f0) at libraries/DAQ/DEVIOWorkerThread.cc:107 #9 0x00002b6827e71070 in ?? () from /lib64/libstdc++.so.6 #10 0x00002b68279a3dd5 in start_thread () from /lib64/libpthread.so.0 #11 0x00002b68286d9ead in clone () from /lib64/libc.so.6 ===========================================================