[2012-01-31 13:37:16.230097] C [client-handshake.c:121:rpc_client_ping_timer_expired] 0-work-client-3: server 10.250.39.13:24011 has not responded in the last 42 seconds, disconnecting.
[2012-01-31 13:37:16.230830] E [rpc-clnt.c:341:saved_frames_unwind] (-->/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fc3a823b568] (-->/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7fc3a823acfd] (-->/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc3a823ac5e]))) 0-work-client-3: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2012-01-31 13:35:54.222786
[2012-01-31 13:37:16.230889] E [client3_1-fops.c:2302:client3_1_readv_cbk] 0-work-client-3: remote operation failed: Transport endpoint is not connected
[2012-01-31 13:37:16.230923] W [fuse-bridge.c:1754:fuse_readv_cbk] 0-glusterfs-fuse: 20795: READ => -1 (Transport endpoint is not connected)
[2012-01-31 13:37:16.230983] E [rpc-clnt.c:341:saved_frames_unwind] (-->/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fc3a823b568] (-->/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7fc3a823acfd] (-->/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc3a823ac5e]))) 0-work-client-3: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2012-01-31 13:35:54.222802
[2012-01-31 13:37:16.230996] E [client3_1-fops.c:2302:client3_1_readv_cbk] 0-work-client-3: remote operation failed: Transport endpoint is not connected
[2012-01-31 13:37:16.231689] I [socket.c:2275:socket_submit_request] 0-work-client-3: not connected (priv->connected = 0)
[2012-01-31 13:37:16.231702] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-work-client-3: failed to submit rpc-request (XID: 0x5259x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (work-client-3)
[2012-01-31 13:37:16.231714] E [client3_1-fops.c:2302:client3_1_readv_cbk] 0-work-client-3: remote operation failed: Transport endpoint is not connected
[2012-01-31 13:37:16.231734] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-work-client-3: failed to submit rpc-request (XID: 0x5260x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (work-client-3)
[2012-01-31 13:37:16.231745] E [client3_1-fops.c:2302:client3_1_readv_cbk] 0-work-client-3: remote operation failed: Transport endpoint is not connected
[2012-01-31 13:37:16.231761] W [fuse-bridge.c:1754:fuse_readv_cbk] 0-glusterfs-fuse: 20812: READ => -1 (Transport endpoint is not connected)
[2012-01-31 13:37:16.231789] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-work-client-3: failed to submit rpc-request (XID: 0x5261x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (work-client-3)
[2012-01-31 13:37:16.231801] E [client3_1-fops.c:2302:client3_1_readv_cbk] 0-work-client-3: remote operation failed: Transport endpoint is not connected
[2012-01-31 13:37:16.231817] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-work-client-3: failed to submit rpc-request (XID: 0x5262x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (work-client-3)
[2012-01-31 13:37:16.231827] E [client3_1-fops.c:2302:client3_1_readv_cbk] 0-work-client-3: remote operation failed: Transport endpoint is not connected
[2012-01-31 13:37:16.231842] W [fuse-bridge.c:1754:fuse_readv_cbk] 0-glusterfs-fuse: 20796: READ => -1 (Transport endpoint is not connected)
[2012-01-31 13:37:16.231884] E [rpc-clnt.c:341:saved_frames_unwind] (-->/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fc3a823b568] (-->/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7fc3a823acfd] (-->/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc3a823ac5e]))) 0-work-client-3: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-01-31 13:36:08.863314
[2012-01-31 13:37:16.231938] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-work-client-3: remote operation failed: Transport endpoint is not connected
[2012-01-31 13:37:16.231961] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-work-client-3: failed to submit rpc-request (XID: 0x5263x Program: GlusterFS 3.1, ProgVers: 310, Proc: 13) to rpc-transport (work-client-3)
[2012-01-31 13:37:16.231973] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-work-client-3: remote operation failed: Transport endpoint is not connected
[2012-01-31 13:37:16.231985] W [client3_1-fops.c:3622:client3_1_writev] 0-work-client-3: failed to send the fop: Stale NFS file handle
pending frames:
frame : type(1) op(STAT)
frame : type(1) op(WRITE)
frame : type(1) op(WRITE)

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2012-01-31 13:37:16
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.2.5
/lib64/libc.so.6(+0x32900)[0x7fc3a7889900]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so(wb_sync_cbk+0x30)[0x7fc3a4342250]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/cluster/distribute.so(dht_writev_cbk+0xda)[0x7fc3a455998a]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/client.so(client3_1_writev+0x13a)[0x7fc3a4795b5a]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/client.so(client_writev+0xa1)[0x7fc3a4785101]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/cluster/distribute.so(dht_writev+0x168)[0x7fc3a455f028]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so(wb_sync+0x4f6)[0x7fc3a433c3e6]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so(wb_do_ops+0x53)[0x7fc3a433f623]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so(wb_process_queue+0xf2)[0x7fc3a433cb72]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so(wb_sync_cbk+0x101)[0x7fc3a4342321]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/cluster/distribute.so(dht_writev_cbk+0xda)[0x7fc3a455998a]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/client.so(client3_1_writev_cbk+0x5d7)[0x7fc3a479e057]
/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(saved_frames_unwind+0x1d2)[0x7fc3a823abc2]
/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fc3a823ac5e]
/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d)[0x7fc3a823acfd]
/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78)[0x7fc3a823b568]
/opt/glusterfs/3.2.5/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7fc3a8236458]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so(socket_event_poll_err+0x54)[0x7fc3a59e4824]
/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so(socket_event_handler+0x12e)[0x7fc3a59eb39e]
/opt/glusterfs/3.2.5/lib64/libglusterfs.so.0(+0x3e744)[0x7fc3a8481744]
/opt/glusterfs/3.2.5/sbin/glusterfs(main+0x2a9)[0x405e99]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fc3a7875cdd]
/opt/glusterfs/3.2.5/sbin/glusterfs[0x4036c9]